Page MenuHomePhabricator
Create Task
Maniphest T88445

MediaWiki active/active datacenter investigation and work (tracking)
Closed, ResolvedPublic
Actions

Description

MediaWiki support for having "reads" (GET/HEAD requests) be routed to multiple datacenters, with writes (POST requests) going to a single datacenter. Having writes be routable to multiple datacenters is not in scope.

More details about what this project entails can be found in the Multi-datacenter RFC.

Details

SubjectRepoBranchLines +/-
[WIP] Added WANObectStash class with cassandra subclassmediawiki/coremaster+283 -0
Added query/connection expectation support to TransactionProfilermediawiki/coremaster+109 -0
Do not call purgeExpiredRestrictions on simple page viewsmediawiki/coremaster+3 -28
Customize query in gerrit

Related Objects
Search...

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusSubtypeAssignedTask
· · ·
ResolvedaaronT88445 MediaWiki active/active datacenter investigation and work (tracking)
ResolvedaaronT88492 Devise caching (memcached) strategy for multi-DC mediawiki
ResolvedaaronT88493 Devise stashing strategy for multi-DC mediawiki
ResolvedaaronT88506 Make FileRepo file cache misses use slaves instead of the master
ResolvedaaronT88634 Make a ReplicatedBagOStuff class for multi-DC use
ResolvedaaronT88635 ObjectCacheSessionHandler should avoid pointless writes in write()
ResolvedaaronT88666 RFC: Master/slave datacenter strategy for MediaWiki
ResolvedaaronT89184 Remove cache anti-dependencies
ResolvedaaronT89308 Create a local job class that enqueues jobs to final location
ResolvedaaronT89389 Try to avoid doCascadeProtectionUpdates() call on page view
ResolvedaaronT91279 Augment user_touched with an in-memory field
ResolvedaaronT91284 Avoid wl_notificationtimestamp updates on page views
ResolvedaaronT91815 Audit cache usage to see what can be DC-local and what needs cross-DC purges
ResolvedaaronT97335 Make CachingSiteStore use WANObjectCache
ResolvedaaronT91819 Use local jobqueue class for jobs enqueued on pages views
ResolvedaaronT91837 Consider moving various DB writes on page views to using local jobs
ResolvedaaronT91869 Implement a replication strategy for Swift
ResolvedaaronT91870 Figure out a replication strategy for ElasticSearch
ResolvedaaronT92007 Investigate a better alternative for making user_newtalk talk updates
OpenNoneT92357 Fix database master queries from HTTP GET/HEAD before active-active multi-dc
DeclinedaaronT93006 Figure out a cross-DC strategy for the BloomFilter caches
ResolvedaaronT93141 Convert various core/extension cache users to the WAN cache as needed
DeclinedaaronT92863 Configure 'enqueue' job queue type to be DC-local and in a dedicated runner loop
ResolvedaaronT93144 Check multi-datacenter compatibility with GettingStarted's use of Redis
ResolvedmatthiasmullieT94028 DB master connections requested by Flow on GET/HEAD requests
ResolvedaaronT94448 Echo doing master queries on simple GET requests
ResolvedaaronT94480 Audit and fix extensions that call saveSettings() to lazy-set preferences when loaded
ResolvedaaronT95126 Make RedisBagOStuff support tag name hashing
ResolvedaaronT97620 Convert various core/extension cache users to ReplicatedBagOStuff
ResolvedaaronT97686 Keep TorExitNodes node list cache populated in *both* datacenters
ResolvedaaronT99208 Make sure $wgUseLocalMessageCache works with multiple DCs
ResolvedaaronT108253 Make sure CentralAuth login tokens work with two datacenters
DeclinedaaronT111575 Make $wgSessionCacheType and $wgMainStash caches multi-DC ready
Resolved• mobrovacT134811 Consider REST with SSL (HyperSwitch/Cassandra) for session storage
ResolvedaaronT112708 Support for master/slave FileBackend distinction in FileBackendMultiwrite
ResolvedaaronT113192 Consider using delayed rebound purges for CDN
ResolvedaaronT113204 Smart caching logic for handling cross-DC network outages
ResolvedaaronT111266 Make LoadBalancer slave lag check and read-only mode more robust (for example, using pt-heartbeat)
OpenNoneT114271 Audit *all* datastores to see how DR failover is handled
ResolvedJoeT114273 Cleanup active-DC based MW config code and make it more robust and easy to change
Resolved• jcrespoT124795 codfw is in read only according to mediawiki
Restricted Task
DuplicateNoneT114398 Make and document DC switch-over scripts
OpenNoneT119641 Split-brain strategy for services that use config managed by etcd
Resolved• jcrespoT119642 Create a Master-master topology between datacenters for easier failover (setup circular replication dallas -> eqiad for mysql databases)
ResolvedaaronT97562 WANObjectCache relay daemon or mcrouter support
ResolvedSBissonT120009 Flow: Use WAN cache delete() and replica-based filling to avoid merge()
ResolvedKrinkleT130946 Make ?action=markpatrolled require POST
Resolved• CatropeT120543 Investigate write mode flag cleanup for Flow API modules
ResolvedaaronT135027 MASTER_POS_WAIT() alternative that works cross-DC
ResolvedaaronT137326 Log post-send DB updates and make sure their frequency is acceptable
ResolvedfgiunchediT127455 Enable HTTPS for Swift traffic
Resolved• CatropeT164860 Update Echo's caching strategy for multi-dc compatibility
Restricted Task
OpenNoneT204245 Run MediaWiki media originals active/active
ResolvedEevansT206016 Create a service for session storage
OpenNoneT219592 Frequent Echo DB_PRIMARY write queries on HTTP GET
DeclinedaaronT221159 FY18/19 TEC1.6 Q4: Improve or replace the usage of GTID_WAIT with pt-heartbeat in MW
ResolvedMarosteguiT220170 Address Database hardware infrastructure blockers on datacenter switchover & multi-dc deployment
ResolvedKrinkleT270223 FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking)
ResolvedtstarlingT91820 Create HTTP verb and sticky cookie DC routing in VCL
ResolvedaaronT91816 Add code to enable setting sticky DC cookies for POST requests
ResolvedaaronT212129 Move MainStash out of Redis to a simpler multi-dc aware solution
ResolvedtstarlingT315271 db1151, db2144 X2 masters error: Could not execute Delete_rows_v1 event on table mainstash.objectstash
ResolvedtstarlingT134809 App servers <=> mariadb SSL/TLS for cross-datacenter writes
· · ·

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Rosencrantz subscribed.May 17 2016, 3:20 PM
Jdforrester-WMF closed subtask T88044: Make rollback use POST instead of GET (use AJAX in GUI) as Resolved.May 23 2016, 9:23 PM
Krinkle reopened subtask T88044: Make rollback use POST instead of GET (use AJAX in GUI) as Open.May 27 2016, 8:03 PM
aaron created subtask T137326: Log post-send DB updates and make sure their frequency is acceptable.Jun 8 2016, 4:47 PM
bd808 unsubscribed.Jun 10 2016, 1:27 AM
aaron closed subtask T137326: Log post-send DB updates and make sure their frequency is acceptable as Resolved.Jun 20 2016, 9:17 PM
aaron removed a subtask: T106386: Compress data at external storage.Aug 3 2016, 5:25 PM
aaron closed subtask T135027: MASTER_POS_WAIT() alternative that works cross-DC as Resolved.Aug 8 2016, 8:18 PM
aaron closed subtask T111266: Make LoadBalancer slave lag check and read-only mode more robust (for example, using pt-heartbeat) as Resolved.
aaron renamed this task from MediaWiki multi-datacenter investigation and work to MediaWiki active/active datacenter investigation and work.Aug 11 2016, 12:10 AM
aaron added a subtask: T127455: Enable HTTPS for Swift traffic.Aug 12 2016, 5:21 AM
BBlack mentioned this in T142848: Stop using persistent storage in our backend varnish layers..Aug 12 2016, 4:50 PM
aaron closed subtask T111575: Make $wgSessionCacheType and $wgMainStash caches multi-DC ready as Declined.Sep 4 2016, 12:33 PM
Liuxinyu970226 subscribed.Oct 3 2016, 2:44 PM
aaron closed subtask T97562: WANObjectCache relay daemon or mcrouter support as Resolved.Oct 11 2016, 8:11 PM
Krinkle closed subtask T130946: Make ?action=markpatrolled require POST as Resolved.Nov 10 2016, 8:39 PM
aaron closed subtask T120009: Flow: Use WAN cache delete() and replica-based filling to avoid merge() as Resolved.Dec 14 2016, 5:29 PM
fgiunchedi closed subtask T127455: Enable HTTPS for Swift traffic as Resolved.Mar 13 2017, 1:33 PM
Krinkle mentioned this in T115890: Reduce and clarify ObjectCache entry points.Mar 14 2017, 1:12 AM
Krinkle mentioned this in Multiple-active-datacenters archived.May 3 2017, 7:34 PM
Krinkle mentioned this in Sustainability (MediaWiki-MultiDC).May 3 2017, 7:56 PM
Krinkle moved this task from Doing to MediaWiki-MultiDC on the Sustainability board.May 3 2017, 8:02 PM
Krinkle edited projects, added Sustainability (MediaWiki-MultiDC); removed Sustainability.
elukey subscribed.May 4 2017, 12:57 PM
• jcrespo added a subtask: T164504: Tracking: Cleanup x1 database connection patterns.May 4 2017, 4:16 PM
Krinkle moved this task from Later to Current: Performance Team on the Sustainability (MediaWiki-MultiDC) board.May 4 2017, 7:40 PM
• Catrope closed subtask T120543: Investigate write mode flag cleanup for Flow API modules as Resolved.May 9 2017, 6:18 PM
• Catrope added a subtask: T164860: Update Echo's caching strategy for multi-dc compatibility.May 9 2017, 6:23 PM
Krinkle changed the status of subtask T88044: Make rollback use POST instead of GET (use AJAX in GUI) from Open to Stalled.May 26 2017, 5:09 PM
• jcrespo mentioned this in T154658: Prepare and improve the datacenter switchover procedure.Jul 25 2017, 4:44 PM
• Imarlier moved this task from Doing (old) to To-do: Goals prioritized current Quarter on the Performance-Team board.Jan 18 2018, 3:56 PM
Krinkle unsubscribed.Jan 23 2018, 12:03 AM
Joe reopened subtask T97562: WANObjectCache relay daemon or mcrouter support as Open.Feb 16 2018, 6:53 AM
Krinkle renamed this task from MediaWiki active/active datacenter investigation and work to MediaWiki active/active datacenter investigation and work (tracking).Jun 20 2018, 9:55 AM
• Imarlier closed subtask T97562: WANObjectCache relay daemon or mcrouter support as Resolved.Jun 20 2018, 10:19 AM
aaron removed a subtask: T164504: Tracking: Cleanup x1 database connection patterns.Aug 3 2018, 6:07 PM
• jcrespo added a subtask: Restricted Task.Aug 7 2018, 7:03 AM
• Imarlier subscribed.Aug 9 2018, 5:15 PM
• Catrope closed subtask T164860: Update Echo's caching strategy for multi-dc compatibility as Resolved.Aug 13 2018, 7:01 PM
CCicalese_WMF closed subtask Restricted Task as Resolved.Nov 6 2018, 2:29 AM
• Imarlier edited projects, added Performance-Team (Radar); removed Performance-Team.Dec 5 2018, 5:07 PM
aaron added a subtask: T212129: Move MainStash out of Redis to a simpler multi-dc aware solution.Dec 18 2018, 8:19 PM
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.Jan 21 2019, 2:17 AM
• mobrovac added projects: Platform Team Legacy (Epic), Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Services (designing), User-mobrovac.Feb 7 2019, 11:19 PM
CDanis subscribed.Feb 8 2019, 12:34 AM
• Mholloway subscribed.Feb 24 2019, 11:21 PM
• mobrovac edited projects, added Platform Engineering (Multi-DC (TEC1)); removed Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1)).Feb 27 2019, 11:29 PM
• mobrovac added a subtask: T206016: Create a service for session storage.Feb 27 2019, 11:34 PM
• mobrovac closed subtask T134811: Consider REST with SSL (HyperSwitch/Cassandra) for session storage as Resolved.
• mobrovac added a subtask: T172497: Fix mediawiki heartbeat model, change pt-heartbeat model to not use super-user, avoid SPOF and switch automatically to the real master without puppet dependency.Apr 15 2019, 8:20 PM
• mobrovac added a subtask: T221159: FY18/19 TEC1.6 Q4: Improve or replace the usage of GTID_WAIT with pt-heartbeat in MW.Apr 16 2019, 11:50 PM
• mobrovac removed a subtask: T172497: Fix mediawiki heartbeat model, change pt-heartbeat model to not use super-user, avoid SPOF and switch automatically to the real master without puppet dependency.
• mobrovac added a subtask: T220170: Address Database hardware infrastructure blockers on datacenter switchover & multi-dc deployment.May 31 2019, 11:47 AM
Krinkle changed the status of subtask T134809: App servers <=> mariadb SSL/TLS for cross-datacenter writes from Open to Stalled.Jul 1 2019, 8:28 PM
• WDoranWMF removed projects: Services (designing), Platform Team Legacy (Epic).Jul 5 2019, 7:31 PM
• WDoranWMF moved this task from Multi-DC (TEC1) to mop on the Platform Engineering board.Jul 26 2019, 6:47 PM
• WDoranWMF edited projects, added Core Platform Team Initiatives (Multi-DC (TEC1)); removed Platform Engineering (Multi-DC (TEC1)).
Marostegui subscribed.Aug 7 2019, 9:52 AM
BPirkle mentioned this in T233963: Add serialization options to RESTBagOStuff.Sep 26 2019, 4:03 PM
Marostegui closed subtask T220170: Address Database hardware infrastructure blockers on datacenter switchover & multi-dc deployment as Resolved.Dec 12 2019, 6:02 AM
CCicalese_WMF added a project: Platform Team Workboards (Initiatives).Mar 24 2020, 10:29 PM
Naike changed the status of subtask T206016: Create a service for session storage from Open to Stalled.May 29 2020, 9:21 AM
Krinkle mentioned this in T254634: Determine and implement multi-dc strategy for ChronologyProtector.Jun 6 2020, 3:42 AM
Aklapper removed aaron as the assignee of this task.Jun 19 2020, 4:26 PM

This task has been assigned to the same task owner for more than two years. Resetting task assignee due to inactivity, to decrease task cookie-licking and to get a slightly more realistic overview of plans. Please feel free to assign this task to yourself again if you still realistically work or plan to work on this task - it would be welcome!

For tips how to manage individual work in Phabricator (noisy notifications, lists of task, etc.), see https://phabricator.wikimedia.org/T228575#6237124 for available options.
(For the records, two emails were sent to assignee addresses before resetting assignees. See T228575 for more info and for potential feedback. Thanks!)

Krinkle removed a subtask: T88044: Make rollback use POST instead of GET (use AJAX in GUI).Jul 2 2020, 4:08 PM
Krinkle changed the status of subtask T212129: Move MainStash out of Redis to a simpler multi-dc aware solution from Open to Stalled.Jul 6 2020, 8:36 PM
• gsingers subscribed.Sep 16 2020, 3:45 PM
LSobanski subscribed.Sep 25 2020, 9:51 AM
jijiki changed the status of subtask T212129: Move MainStash out of Redis to a simpler multi-dc aware solution from Stalled to Open.Dec 14 2020, 1:14 PM
Krinkle moved this task from Current: Performance Team to Later on the Sustainability (MediaWiki-MultiDC) board.Dec 22 2020, 10:41 PM
aaron closed subtask T221159: FY18/19 TEC1.6 Q4: Improve or replace the usage of GTID_WAIT with pt-heartbeat in MW as Declined.Jan 12 2021, 5:20 AM
RhinosF1 subscribed.Jan 12 2021, 11:22 AM
taavi subscribed.Feb 12 2021, 1:33 PM
Izno subscribed.Aug 5 2021, 5:50 AM
kostajh mentioned this in T288948: Consider codifying whether a given hook allows making primary db reads/writes.Aug 16 2021, 12:40 PM
greg closed subtask T206016: Create a service for session storage as Resolved.Oct 20 2021, 12:06 AM
• Mholloway unsubscribed.Oct 20 2021, 12:07 AM
Krinkle added a subtask: T270225: Finish session storage to actually meet multi-DC requirements.Oct 20 2021, 1:06 AM
Krinkle removed a subtask: T270225: Finish session storage to actually meet multi-DC requirements.
Krinkle added a subtask: T270223: FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking).
Urbanecm_WMF changed the status of subtask T219592: Frequent Echo DB_PRIMARY write queries on HTTP GET from Open to In Progress.Jan 31 2022, 11:34 AM
tstarling changed the status of subtask T134809: App servers <=> mariadb SSL/TLS for cross-datacenter writes from Stalled to Open.May 26 2022, 4:32 AM
tstarling closed subtask T134809: App servers <=> mariadb SSL/TLS for cross-datacenter writes as Resolved.Jun 14 2022, 12:00 AM
tstarling closed subtask T212129: Move MainStash out of Redis to a simpler multi-dc aware solution as Resolved.Jun 17 2022, 1:38 AM
Krinkle closed subtask T91820: Create HTTP verb and sticky cookie DC routing in VCL as Resolved.Jul 6 2022, 12:57 AM
Dzahn added a subtask: T315271: db1151, db2144 X2 masters error: Could not execute Delete_rows_v1 event on table mainstash.objectstash.Aug 16 2022, 4:07 AM
Krinkle closed subtask T315271: db1151, db2144 X2 masters error: Could not execute Delete_rows_v1 event on table mainstash.objectstash as Resolved.Aug 22 2022, 11:11 PM
kostajh subscribed.Sep 22 2022, 10:42 AM
Krinkle closed subtask T270223: FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking) as Resolved.Sep 26 2022, 4:49 PM
Krinkle removed a project: Performance-Team (Radar).Aug 6 2023, 10:23 PM
Krinkle removed subscribers: • gsingers, • Imarlier, • MZMcBride and 3 others.
Aklapper added a comment.Oct 17 2023, 7:44 AM

Does this task serve any purpose in itself that is not covered by the Sustainability (MediaWiki-MultiDC) project tag?

aaron closed this task as Resolved.Oct 17 2023, 7:42 PM
aaron claimed this task.

At this point, I don't think there is much value beyond the few remaining open tasks.

Pppery changed the status of subtask T219592: Frequent Echo DB_PRIMARY write queries on HTTP GET from In Progress to Open.Sun, Nov 10, 9:13 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits