[xingzihai]

Summary

fixCharset with DetectCharset() relied solely on chardet's statistical detection when the Content-Type header has no charset. chardet ignores the charset declared in <meta http-equiv="Content-Type">, so pages served as e.g. windows-1251 that declare their encoding only via the meta tag are misdetected and misdecoded (see #777, where a windows-1251 page renders as ÑÄÎ «Ïðîìåòåé»).

Fix

In Response.fixCharset, when the Content-Type header has no charset, first consult golang.org/x/net/html/charset.DetermineEncoding, which inspects the BOM, then the HTML <meta> charset declaration, before falling back to its own detection. chardet is kept as a fallback when DetermineEncoding returns no name, so the previous behavior is preserved for bodies without any declaration.

if _, name, _ := charset.DetermineEncoding(r.Body, contentType); name != "" {
    contentType = "text/plain; charset=" + name
} else {
    d := chardet.NewTextDetector()
    det, err := d.DetectBest(r.Body)
    if err != nil {
        return err
    }
    contentType = "text/plain; charset=" + det.Charset
}

Testing

New regression test TestResponseFixCharsetMetaDeclaration in response_test.go: a windows-1251 body that declares charset=windows-1251 only via <meta>, with a Content-Type: text/html header (no charset). Before the fix the test fails (body contains mojibake ÐØÕØÝåéì); after the fix it passes (body contains the proper UTF-8 "Прометей").

• go test . -run TestResponseFixCharset -count=1 -v → PASS
• go test -count=1 ./... → all packages green
• gofmt -l . and go vet ./... → clean

Fixes #777