[pr] Feature/high reliability #36

stephen-palmer · 2018-05-15T13:53:00Z

Added new feature, High Reliablity mode. For details see the documentation added to the README.md file in this PR.

A few supporting refactors are included here:

Moved the loki.js database handling to the base Cache class (from CacheRAM), so now any cache module can use the in-memory DB for storage. This was necessary to give CacheFS a facility for tracking reliability factors for versions
Added a small utility class to handle cluster messaging, to facilitate worker nodes reading and writing to the in-memory database on the master node. This messaging class doesn't have test coverage yet - I haven't cracked the nut on properly testing forked clients with Mocha (and getting code coverage with Istanbul).
Consolidated all of the string literals for file types into constants

…e uploads (configured via reliabilityThreshold) of the same file to have a matching content hash before the file will be cached and served.

- Added requireUniqueClient feature to reliability_manager, where transactions for the same version are forced to alternate between different clients to increment the reliability index.

…ences to a new FILE_TYPE constant. This also has the nice side effect of making it simpler to add new types, if desired in the future.

coveralls · 2018-05-15T13:55:24Z

Coverage decreased (-1.8%) to 90.353% when pulling a461810 on feature/high-reliability into 5c662b8 on master.

pullrequest

My first pass at this looked pretty solid. Have a few recommendations regarding optimization of multiple async calls.

Was this helpful? Let us know!

pullrequest · 2018-05-15T16:25:24Z

lib/cache/reliability_manager.js

+            }
+        }
+
+        cm.listenFor("_updateReliabilityFactorForVersion", async (data) => {


Using async function with no await.

pullrequest · 2018-05-15T16:25:24Z

lib/cache/cache_fs.js

@@ -264,6 +289,39 @@ class PutTransactionFS extends PutTransaction {

        return new Promise(resolve => stream.on('open', () => resolve(stream)));
    }
+
+    async writeFilesToPath(filePath) {
+        const paths = [];


This could utilize Promise.all for a more parallel operation with less upfront allocation and moving parts:

await fs.ensureDir(filePath); return Promise.all(this._files.map(async f => { const dest = `${path.join(filePath, f.byteHash.toString('hex'))}.${f.type}`; await fs.copyFile(f.file, dest); return dest; }));

pullrequest · 2018-05-15T16:25:24Z

lib/cache/cache_fs.js

-
-        await transaction.finalize();
-        return Promise.all(transaction.files.map(moveFile));
+        for(const file of transaction.files) {


Just wanted to verify that you intend for this cache to be written to serially. Using for/of with a blocked await will perform these actions serially, while using Promise.all would give you an eager parallel, provided your operation could support it:

await Promise.all( transaction.files.map(async file => { try { const filePath = await self._writeFileToCache(file.type, transaction.guid, transaction.hash, file.file); helpers.log(consts.LOG_TEST, `Added file to cache: ${file.size} ${filePath}`); } catch (err) { helpers.log(consts.LOG_DBG, err); } }) );

pullrequest · 2018-05-15T16:25:25Z

lib/cache/cache_fs.js

@@ -153,10 +159,20 @@ class CacheFS extends CacheBase {
            self.emit('cleanup_search_finish', progressData());

            for(const d of deleteItems) {
-                self.emit('cleanup_delete_item', d.path);
+                const guidHash = CacheFS._extractGuidAndHashFromFilepath(d.path);


I know this is an existing loop, but had the same question about serial vs. parallel async operations here.

good call ✅

pullrequest · 2018-05-15T16:25:25Z

lib/cache/cache_fs.js

@@ -249,12 +274,12 @@ class PutTransactionFS extends PutTransaction {
            throw new Error("Invalid size for write stream");
        }

-        if(type !== 'a' && type !== 'i' && type !== 'r') {
+        if(Object.values(consts.FILE_TYPE).indexOf(type) < 0) {


You can use includes for a more succinct description of intent:

if(!Object.values(consts.FILE_TYPE).includes(type)) {

pullrequest · 2018-05-15T16:25:25Z

lib/cache/reliability_manager.js

+     * @param {string} guidStr
+     * @param {string} hashStr
+     * @param {boolean} create
+     * @returns {Object}


Can this also return null since you do a !entry check?

yes, it can - I rely on that for tests. Accordingly I'll add null as a return type.

pullrequest · 2018-05-15T16:25:25Z

test/cache_fs.js

+            });
+
+            it("should remove versions from the reliability manager, when in high reliability mode", async () => {
+                const opts = Object.assign({}, cacheOpts);


A random tip: your Node.js engines support object rest/spread natively, so you could take advantage of that in your codebase if you feel inclined:

const opts = { ...cacheOpts };

oh, cool .. I'll leave as is but this is a cool trick!

pullrequest · 2018-05-15T16:25:25Z

lib/cache/cache_fs.js

@@ -237,6 +254,14 @@ class PutTransactionFS extends PutTransaction {
        return this._files;
    }

+    async invalidate() {
+        await super.invalidate();
+        for(const f of this._files)


I think this should support a parallel operation with Promise.all:

await super.invalidate(); await Promise.all(this._files.map(f => fs.unlink(f.file))); this._files = [];

pullrequest · 2018-05-15T16:25:25Z

lib/cache/cache_base.js

+    /**
+     *
+     */
+    async _saveDb() {


Note: this function does not do any awaiting, so you can technically remove the async keyword.

pullrequest · 2018-05-15T16:25:26Z

lib/cache/cache_ram.js

@@ -396,38 +372,81 @@ class PutTransactionRAM extends PutTransaction {
            throw new Error("Invalid size for write stream");
        }

-        if(type !== 'a' && type !== 'i' && type !== 'r') {
+        if(Object.values(consts.FILE_TYPE).indexOf(type) < 0) {


I would recommend using includes so the intent is clearer:

if(!Object.values(consts.FILE_TYPE).includes(type)) {

…e some parallelization of async operations that were previously done in serial.

BrettKercher

Couple questions about the multiClient option, but looks good everywhere else!

BrettKercher · 2018-05-17T16:20:17Z

lib/cache/reliability_manager.js

+        const entry = this.getEntry(params.guidStr, params.hashStr, true);
+        if(!entry.versionHash) {
+            entry.versionHash = params.versionHashStr;
+            entry.clientId = params.clientId;


If the entry's clientId is set when it's first created, it looks like it will always treat the first transaction as a duplicate when multiClient is enabled

I think you're right, will look closer at this.

BrettKercher · 2018-05-17T16:30:13Z

lib/cache/reliability_manager.js

+            return entry;
+        }
+
+        entry.clientId = params.clientId;


With two clients "A" and "B" if transactions come in the order A->B->A->B, it would end up with a reliabilityFactor of 4, but transactions in the order A->A->B->B, would have a reliabilityFactor of 2. Seems a little odd that the same transition set can end up with differing reliabilityFactors if the timing changes a bit. Is that intended?

Yes, this is intended - I don't want to actually track unique clients. In practice, I believe 99% of the time the default threshold of 2 will be used, so we would hardly gain from the added complexity of keeping track of all unique clients.

BrettKercher · 2018-05-17T16:32:06Z

test/reliability_manager.js

+            });
+        });
+
+        describe("multiClient", () => {


It looks like this is testing transactions from 3 different client addresses - "A", "B", and "".

good catch - this test as written was actually masking the bug you found where the clientId was being set on a new entry.

…n incorrectly that would have caught the bug to begin with.

pullrequest

	1 Message
⚠️	Due to its size, this pull request will likely have a little longer turnaround time and will probably require multiple passes from our reviewers.

stephen-palmer added 11 commits May 4, 2018 20:43

New feature: reliabilityFactor/reliabilityThreshold. Requires multipl…

836d72a

…e uploads (configured via reliabilityThreshold) of the same file to have a matching content hash before the file will be cached and served.

- Added tests and did some refactoring/refinement

2469d6a

- Added requireUniqueClient feature to reliability_manager, where transactions for the same version are forced to alternate between different clients to increment the reliability index.

Version bump

72c0b2c

Updated mocha to latest version

7e4c634

Renamed ‘requireUniqueClients’ option to ‘multiClient’

4e7edc6

Added documentation

000e287

doc edits

fa09a3b

Only process valid transactions with the reliability manager

54c26a2

Automatically invalidate transactions with zero-value asset hashes

73993dd

Long overdue refactor to replace file type string literals with refer…

f3ddf49

…ences to a new FILE_TYPE constant. This also has the nice side effect of making it simpler to add new types, if desired in the future.

Exclude info file contents from determinism calculation

95cafaa

pullrequest bot reviewed May 15, 2018

View reviewed changes

Code cleanup based on PR feedback. The only functional differences ar…

7f3e061

…e some parallelization of async operations that were previously done in serial.

stephen-palmer requested a review from BrettKercher May 16, 2018 13:12

Fix error logging of GUID

cb93af7

BrettKercher reviewed May 17, 2018

View reviewed changes

stephen-palmer added 2 commits May 17, 2018 13:39

Fixed bug with multiClient option, and fixed the test that was writte…

6ee0d83

…n incorrectly that would have caught the bug to begin with.

Documentation edits for High Reliability Mode.

a461810

stephen-palmer merged commit bf012ab into master Jun 7, 2018

stephen-palmer deleted the feature/high-reliability branch December 10, 2018 19:06

pullrequest bot reviewed Oct 28, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pr] Feature/high reliability #36

[pr] Feature/high reliability #36

stephen-palmer commented May 15, 2018

coveralls commented May 15, 2018 •

edited

Loading

pullrequest bot left a comment

pullrequest bot May 15, 2018

stephen-palmer May 15, 2018

pullrequest bot May 15, 2018

stephen-palmer May 15, 2018

pullrequest bot May 15, 2018

stephen-palmer May 15, 2018

pullrequest bot May 15, 2018

stephen-palmer May 15, 2018

pullrequest bot May 15, 2018

stephen-palmer May 15, 2018

pullrequest bot May 15, 2018

stephen-palmer May 15, 2018

pullrequest bot May 15, 2018

stephen-palmer May 15, 2018

pullrequest bot May 15, 2018

stephen-palmer May 15, 2018

pullrequest bot May 15, 2018

stephen-palmer May 15, 2018

pullrequest bot May 15, 2018

stephen-palmer May 15, 2018

BrettKercher left a comment

BrettKercher May 17, 2018

stephen-palmer May 17, 2018

BrettKercher May 17, 2018

stephen-palmer May 17, 2018

BrettKercher May 17, 2018

stephen-palmer May 17, 2018

pullrequest bot left a comment

[pr] Feature/high reliability #36

[pr] Feature/high reliability #36

Conversation

stephen-palmer commented May 15, 2018

coveralls commented May 15, 2018 • edited Loading

pullrequest bot left a comment

Choose a reason for hiding this comment

pullrequest bot May 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pullrequest bot May 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pullrequest bot May 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pullrequest bot May 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pullrequest bot May 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pullrequest bot May 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pullrequest bot May 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pullrequest bot May 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pullrequest bot May 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pullrequest bot May 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BrettKercher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pullrequest bot left a comment

Choose a reason for hiding this comment

coveralls commented May 15, 2018 •

edited

Loading