日期:2014-05-17  浏览次数:20671 次

lexer html解析一个js过滤的改进

问题描述,使用htmlparser的lexer解析器进行页面解析时发现类似如下的页面会有问题:

?

?

<script>
for(i=0;i<a;i++){

}
</script>

解析后代码变成了:
<script>
for(i=0;i<a;i++){

}
></script>

?

?通过lexer代码发现,实际上只要js代码改成:

<script>
<!--
for(i=0;i<a;i++){

}
-->
</script>

?就不会有问题了,从代码中主要发现它的解析其实没有问题,主要是我们平时的页面规范做的不好,它在解析时会看到字符解析时发现<后面如果有字母就认为它是一个tag:

	protected Node parseString(int start, boolean quotesmart)
			throws ParserException {
		boolean done;
		char ch;
		char quote;

		done = false;
		quote = 0;
		while (!done) {
			ch = mPage.getCharacter(mCursor);
			if (Page.EOF == ch)
				done = true;
			else if (0x1b == ch) // escape
			{
				ch = mPage.getCharacter(mCursor);
				if (Page.EOF == ch)
					done = true;
				else if ('$' == ch) {
					ch = mPage.getCharacter(mCursor);
					if (Page.EOF == ch)
						done = true;
					// JIS X 0208-1978 and JIS X 0208-1983
					else if ('@' == ch || 'B' == ch)
						scanJIS(mCursor);
					/*
					 * // JIS X 0212-1990 else if ('(' == ch) { ch =
					 * mPage.getCharacter (mCursor); if (Page.EOF == ch) done =
					 * true; else if ('D' == ch) scanJIS (mCursor); else {
					 * mPage.ungetCharacter (mCursor); mPage.ungetCharacter
					 * (mCursor); mPage.ungetCharacter (mCursor); } }
					 */
					else {
						mPage.ungetCharacter(mCursor);
						mPage.ungetCharacter(mCursor);
					}
				} else
					mPage.ungetCharacter(mCursor);
			} else if (quotesmart && (0 == quote)
					&& (('\'' == ch) || ('"' == ch)))
				quote = ch; // enter quoted state
			// patch from Gernot Fricke to handle escaped closing quote
			else if (quotesmart && (0 != quote) && ('\\' == ch)) {
				ch = mPage.getCharacter(mCursor); // try to consume escape
				if ((Page.EOF != ch) && ('\\' != ch) // escaped backslash
						&& (ch != quote)) // escaped quote character
					// ( reflects ["] or ['] whichever opened the quotation)
					mPage.ungetCharacter(mCursor); // unconsume char if char not
													// an escape
			} else if (quotesmart && (ch == quote))
				quote = 0; // exit quoted state
			else if (quotesmart && (0 == quote) && (ch == '/')) {
				// handle multiline and double slash comments (with a quote)
				// in script like:
				// I can't handle single quotations.
				ch = mPage.getCharacter(mCursor);
				if (Page.EOF == ch)
					done = true;
				else if ('/' == ch) {
					do
						ch = mPage.getCharacter(mCursor);
					while ((Page.EOF != ch) && ('\n' != ch));
				} else if ('*' == ch) {
					do {
						do
							ch = mPage.getCharacter(mCursor);
						while ((Page.EOF != ch) && ('*' != ch));
						ch = mPage.getCharacter(mCursor);
						if (ch == '*')
							mPage.ungetCharacter(mCursor);
					} while ((Page.EOF != ch) && ('/' != ch));
				} else
					mPage.ungetCharacter(mCursor);
			} else if ((0 == quote) && ('<' == ch)) {
				ch = mPage.getCharacter(mCursor);
				if (Page.EOF == ch)
					done = true;
				// the order of these tests might be optimized for speed:
				else if ('/' == ch
						|| Character.isLetter(ch)
						|| '!' == ch || '%' == ch || '?' == ch) {
					done = true;
					mPage.ungetCharacter(mCursor);
					mPage.ungetCharacter(mCursor);
				} else {
					// it's not a tag, so keep going, but check for quotes
					mPage.ungetCharacter(mCursor);
				}
			}
		}

		return (makeString(start, mCursor.getPosition()));
	}

?因此为了解决这个问题,现在要在上面做一个手脚:

首先在类中间增加了一个标记,script

这个标记是修改了nexNode方法,在返回前判断下是否前一个标签是<script> 或者</script>

然后在parseString中修改其解析方法就可以了,下面是完整的代码:

import java.net.URLConnection;

import org.htmlparser.Node;
import org.htmlparser.lexer.Lexer;
import org.htmlparser.lexer.Page;
import or