Improve UI5 performance for large codebases

[lcartey]

We observed high memory usage and slow evaluation on codebases with a large number of files and string literals.

0f554bd - addresses a join order problem when determining whether elements occur in the same web-app. We prefer to apply the inSameWebapp condition after computing all other conditions, to avoid a cross-product on data flow nodes in the same web app.

ed87aa9 - the BindingStringParser had performance issues on large string literals with a large number of tokens. This was because calculating token operations such as the next token, token containment and the first token created a cross-product on all tokens in the given string, before applying further conditions.

We avoid this problem by creating a tokenOrdering predicate that provides an ordering over all the tokens in the string. This is then used to implement the token operations more efficiently. This approach was complicated by the fact that we can have overlapping and contained tokens. Longer term, I think we would prefer to eliminate this overlap to simplify this code.

We observed this performance issue on a codebase with a large number of *.chunk.js files (which are generated by webpack code splitting). Unfortunately, these files contain a lot of large string literals containing JSON, which we currently pick up as a potential binding string. As a future improvement it would be better if we could exclude these from the binding string calculation all together, perhaps by using data flow to determine whether a string literal flows to a relevant API.

[jeongsoolee09]

Left a question, but otherwise I think it's a brilliant solution to work around the joining issue!

[lcartey]

I'm not very sure if getNextAfterEnd should be used in place of getNext. t.getNextAfterEnd finds a token which may come after a whitespace token that this token may contain, so this may give bar when token t's value is "foo bar", whereas the expected behavior of getNextSkippingWhitespace is to get the token that comes after the entire "foo bar".

@jeongsoolee09 Resurrecting your question, as it got lost.

getNextAfterEnd(..) should return the same set as the previous implementation of getNext(..). In both cases, it will return tokens which begin the character after the previous token ends.

The new implementation of getNext reports the next token according to the beginning position of the token. This will mean it reports tokens that are contained within the current token. That's not what we want for getNextSkippingWhitespace - we want to match the existing behaviour.

---

Note, I've also updated to address some minor inconsistencies with the previous implementation.

[jeongsoolee09]

Makes sense now and completely understood. Thank you!