Page MenuHomePhabricator
Create Task
Maniphest T236904

Improve message group selector performance reducing Translate API request time
Closed, ResolvedPublic
Actions

Description

Hoping it’s not already suggested, nor a T160565 duplicate.

The following may list a (too) large amount of messageGroups:
Special:PageTranslation, Special:AggregateGroups and group choosing tool in header of Special:Translate (following API query)

action=query
format=json
meta=messagegroups
mgformat=tree
mgiconsize=32
mgprop=id|label|icon|priority|prioritylangs|priorityforce

I suppose all these tools use the same internal API.

I hope pagination may improve both server-side and client-side performances (which are currently very poor).

Details

SubjectRepoBranchLines +/-
Alternative TranslateMetadata::preloadGroups() optimizationmediawiki/extensions/Translatemaster+8 -3
Optimize Translate::preloadMetadatamediawiki/extensions/Translatemaster+7 -1
Customize query in gerrit

Related Objects

Event Timeline

Pols12 created this task.Oct 30 2019, 1:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 30 2019, 1:55 PM
Nikerabbit subscribed.Oct 30 2019, 3:13 PM

Speaking of Translate itself, pagination won't help unless the group selector learns to do server side search filtering.

Have you done profiling why the performance is poor?

Special:PageTranslation does not use API queries to my knowledge.

Nikerabbit added a comment.Oct 30 2019, 3:17 PM

For Special:AggregateGroups I suggest to use T90511: Special:AggregateGroups is too large, needs collapsibility.

We could dedicate this ticket to optimizing the message group selector. Is that okay to you?

Pols12 added a comment.Oct 31 2019, 11:28 AM
In T236904#5619674, @Nikerabbit wrote:

For Special:AggregateGroups I suggest to use T90511: Special:AggregateGroups is too large, needs collapsibility.

We could dedicate this ticket to optimizing the message group selector. Is that okay to you?

That’s fine to me, thanks.

In T236904#5619648, @Nikerabbit wrote:

Speaking of Translate itself, pagination won't help unless the group selector learns to do server side search filtering.

Have you done profiling why the performance is poor?

XHR take 27s to get an response (when an response is received…)
Please find a HAR file: https://framadrop.org/r/MuseyfK8RL#QPt9985xVlXfQNMhooZ3g/+fq1zJY1048rQXuYw64MA=
And a performance profile generated by Firefox: https://framadrop.org/r/DTa5JKVB6A#3uQAeYKdSVIPLWVw7ricSHt39XC7xTzog1ujlPJ6zJI=
(sorry, Phabricator refuses I upload them here)

I’m not able to analyze them, however the XHR to API time is really too long to make the tool usable on meta…

Nikerabbit added a comment.Oct 31 2019, 12:29 PM

I can confirm https://meta.wikimedia.org/w/api.php?action=query&format=json&meta=messagegroups&mgformat=tree&mgprop=id%7Clabel%7Cicon%7Cpriority%7Cprioritylangs%7Cpriorityforce&mgiconsize=32 takes >20 seconds.

I tried to have a look what XHGui says about the profile, but it is broken: T237002: XHGui is broken: unrecognized field "cursor.

Nikerabbit triaged this task as High priority.Oct 31 2019, 2:57 PM
Nikerabbit moved this task from Backlog to maintenance and operational issues on the MediaWiki-extensions-Translate board.
Nikerabbit added a subscriber: abi_.

Apparently it works after all: https://performance.wikimedia.org/xhgui/run/view?id=5dbad2613f3dfa7c5f89a004

Something weird going on TranslateMetadata::preloadGroups

Nikerabbit added a comment.Oct 31 2019, 3:04 PM

Plain select * from translate_metadata is pretty fast: 26555 rows in set (0.04 sec).

It's more likely that the frequent calls (18197 to be exact) that run array_diff and array_fill_keys is burning CPU time. I think the preloading is not working properly, as I would expect much less calls to it.

gerritbot added a comment.Oct 31 2019, 6:22 PM

Change 547615 had a related patch set uploaded (by Nikerabbit; owner: Nikerabbit):
[mediawiki/extensions/Translate@master] Optimize Translate::preloadMetadata

https://gerrit.wikimedia.org/r/547615

gerritbot added a project: Patch-For-Review.Oct 31 2019, 6:22 PM
Nikerabbit claimed this task.Oct 31 2019, 6:22 PM
Nikerabbit added a project: Language-Team (Language-2019-October-December).
Nikerabbit added a project: Unplanned-Sprint-Work.
Nikerabbit moved this task from Backlog to In Review on the Language-Team (Language-2019-October-December) board.
gerritbot added a comment.Oct 31 2019, 6:57 PM

Change 547621 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/Translate@master] Optimize TranslateMetadata::preloadGroups() further

https://gerrit.wikimedia.org/r/547621

gerritbot added a comment.Nov 1 2019, 3:43 PM

Change 547615 abandoned by Nikerabbit:
Optimize Translate::preloadMetadata

Reason:
Going for the competing patch.

https://gerrit.wikimedia.org/r/547615

gerritbot added a comment.Nov 1 2019, 3:53 PM

Change 547621 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] Alternative TranslateMetadata::preloadGroups() optimization

https://gerrit.wikimedia.org/r/547621

Maintenance_bot removed a project: Patch-For-Review.Nov 1 2019, 4:10 PM
ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.5; 2019-11-05).Nov 1 2019, 6:00 PM
Nikerabbit moved this task from In Review to Recheck after deployment on the Language-Team (Language-2019-October-December) board.Nov 4 2019, 8:03 AM
Nikerabbit closed this task as Resolved.Dec 11 2019, 8:43 AM
Nikerabbit moved this task from Recheck after deployment to Done on the Language-Team (Language-2019-October-December) board.

The request now takes a few seconds and no obvious hot spots:

FunctionCall CountSelf Wall TimeSelf CPUSelf Memory UsageSelf Peak Memory UsageInclusive Wall TimeInclusive CPUInclusive Memory UsageInclusive Peak Memory Usage
WikiPageMessageGroup::getLabel88508188,192 µs184,315 µs384 bytes168 bytes306,246 µs305,565 µs640 bytes384 bytes
MessageGroups::groupLabelSort73989146,674 µs145,319 µs256 bytes128 bytes476,133 µs475,597 µs2,520 bytes35,952 bytes
Wikimedia\Rdbms\DatabaseMysqlBase::fetchObject28342136,747 µs131,933 µs256 bytes0 bytes341,456 µs342,611 µs15,713,072 bytes13,030,120 bytes
Wikimedia\Rdbms\DatabaseMysqli::doQuery12131,578 µs29,887 µs4,771,976 bytes3,356,448 bytes131,757 µs30,069 µs4,772,488 bytes3,356,480 bytes
TranslateMetadata::preloadGroups18345116,983 µs114,734 µs-13,151,632 bytes1,179,176 bytes721,187 µs690,707 µs6,652,304 bytes17,582,128 bytes
MediaWikiTitleCodec::splitTitleString5522110,986 µs103,348 µs2,516,480 bytes2,783,424 bytes303,451 µs295,144 µs2,555,424 bytes3,280,024 bytes
ApiResult::validateValue@33883583,418 µs82,092 µs264,760 bytes0 bytes313,639 µs314,235 µs1,852,424 bytes0 bytes
ApiResult::isMetadataKey12229883,387 µs86,521 µs640 bytes1,696 bytes83,387 µs86,521 µs640 bytes1,696 bytes
UtfNormal\Validator::cleanUp3420682,315 µs83,814 µs1,930,744 bytes1,048 bytes123,655 µs126,429 µs1,930,872 bytes1,048 bytes

New run: https://performance.wikimedia.org/xhgui/run/view?id=5df0a9a13f3dfa4e51bf02a1
Comparison: https://performance.wikimedia.org/xhgui/run/compare?base=5dbad2613f3dfa7c5f89a004&head=5df0a9a13f3dfa4e51bf02a1

This is definitely fixed.

Pols12 renamed this task from Paginate group getting through Translate API for performance purposes to Improve message group selector performance reducing Translate API request time.Dec 11 2019, 12:20 PM
Pols12 awarded a token.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits