jsoncpp-devel Mailing List for json-cpp (Page 3)

On Mon, May 23, 2011 at 1:50 PM, Baptiste Lepilleur <
bap...@gm...> wrote:

> I propose making the following changes:
> 1) isUInt() returns true only asUInt() can succeed. Similar changes for
> isInt().
>     I think this contract should be generalized. e.g. asUInt() should
> succeed if the value is a positive signed int Value.
> 2) Adds new member functions:
>    isUInt64() that returns true only if asUInt64() can succeed.
>    and isInt64()...
>
> If I understood your problem correctly, this should solve your backward
> compatibility issue. Is that correct?
>

My proposal would be to remove the int/uint distinction entirely, since
JavaScript and JSON do not distinguish between the two. In my own JSON code
i simply use int64_t for all integers and double for doubles (though JS
there's just a single Number type, if i'm not mistaken). In my own use of
json libs and JS engines (SpiderMonkey, QtScript, and Google v8), i've
always found the distinction between signed/unsigned to be philosophically
unsettling because JS doesn't natively have unsigned numbers.

JSON, in fact, does not specify the precision of numeric values (
http://www.ietf.org/rfc/rfc4627.txt?number=4627), probably because it would
be impossible to guaranty that any given implementation (in umpteen[1]
different programming languages) can honor that, e.g. a certain embedded
environment might not have integers >32 bits. Section 2.4 of the JSON RFC
notably does not specify any limits on numbers, so 2^75 (when written in
expanded integer form) is syntactically legal, according to the grammar, but
is almost certainly semantically illegal in any modern computer.

These pages:

http://www.hunlock.com/blogs/The_Complete_Javascript_Number_Reference
http://www.jibbering.com/faq/#FAQ4_7

say that in JS integers are reliable up to "15-16" digits (53 bits). Thus
int64_t can legally hold any JS-legal value.

Obviously, JSON is _not_ only for JavaScript, but i'm using JS as the
baseline here because (A) JSON derives from JS and (B) it is primarily
consumed by JS applications (though it is primarily generated by other
technologies).


[1] = American slang for "very many"

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/

I think you put your finger on something missing in the API.

I propose making the following changes:
1) isUInt() returns true only asUInt() can succeed. Similar changes for
isInt().
    I think this contract should be generalized. e.g. asUInt() should
succeed if the value is a positive signed int Value.
2) Adds new member functions:
   isUInt64() that returns true only if asUInt64() can succeed.
   and isInt64()...

If I understood your problem correctly, this should solve your backward
compatibility issue. Is that correct?

What do you think of this proposal?

Baptiste.

2011/5/23 Aaron Jacobs <ja...@go...>

> Hi all,
>
> I'm working on updating Google's internal version of jsoncpp from an
> ancient
> commit to the most recent version (SVN rev 192), but having some trouble. I
> think I've run into a backwards incompatible change that makes it nearly
> impossible for us to upgrade.
>
> In particular, there's jsoncpp-using code that looks something like this:
>
>    Json::Value value = ...
>    unsigned int value =
>        value.isUInt() ? value.asUInt() :
>            (value.isDouble() ? value.asDouble() : kError);
>
> That is, the code uses the isFoo methods to decide which asFoo method to
> call.
> The problem is that at SVN rev 192, values parsed as integers greater than
> 2^32 will return true for isUInt(), but have an assertion failure for
> asUInt():
>
>    value_.uint_ <= maxUInt
>    "unsigned integer out of UInt range"
>
> You can see this by parsing the string "31121983701778432", for example.
>
> As far as I can tell, there's no way for me to update to the latest version
> of
> jsoncpp without either changing the code above or breaking it (introducing
> incorrectness or a crasher bug into production). Updating the code is not
> an
> option for me since there is too much of it.
>
> Do you have any advice? In particular, am I missing something in the
> Json::Value API?
>
> Thanks,
> Aaron
>

Hi all,

I'm working on updating Google's internal version of jsoncpp from an ancient
commit to the most recent version (SVN rev 192), but having some trouble. I
think I've run into a backwards incompatible change that makes it nearly
impossible for us to upgrade.

In particular, there's jsoncpp-using code that looks something like this:

    Json::Value value = ...
    unsigned int value =
        value.isUInt() ? value.asUInt() :
            (value.isDouble() ? value.asDouble() : kError);

That is, the code uses the isFoo methods to decide which asFoo method to call.
The problem is that at SVN rev 192, values parsed as integers greater than
2^32 will return true for isUInt(), but have an assertion failure for
asUInt():

    value_.uint_ <= maxUInt
    "unsigned integer out of UInt range"

You can see this by parsing the string "31121983701778432", for example.

As far as I can tell, there's no way for me to update to the latest version of
jsoncpp without either changing the code above or breaking it (introducing
incorrectness or a crasher bug into production). Updating the code is not an
option for me since there is too much of it.

Do you have any advice? In particular, am I missing something in the
Json::Value API?

Thanks,
Aaron

On Fri, May 20, 2011 at 4:01 PM, Baptiste Lepilleur <
bap...@gm...> wrote:

> The easiest for you would be to ensure that all your strings are encoded
> using UTF-8. This is the standard way of storing unicode string as
> std::string / const char *.


i can highly recommend this library for doing the conversions:

http://utfcpp.sourceforge.net/

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/

The easiest for you would be to ensure that all your strings are encoded
using UTF-8. This is the standard way of storing unicode string as
std::string / const char *.

Baptiste.

2011/5/18 Сухенький Серафим <ad...@tr...>

> Hi!
>
> I'm using json-cpp as an embedded database driver in my project.
> Now I need to emplement unicode support.
>
> Could you please tell, how can I store unicode strings in Value and get
> them back?
>
> Thanks!
>
>

Hi!

I'm using json-cpp as an embedded database driver in my project.
Now I need to emplement unicode support.

Could you please tell, how can I store unicode strings in Value and get 
them back?

Thanks!

Hi,
I am trying to use the library on a Mac. Since it is not available via
MacPorts I am trying to build it from source.

I've downloaded the tarball ( 0.5.0 ) , and issued :
scons platform=linux-gcc

which compiles fine.

But there is no installation script via scons, or I am not aware of one.
I have tried manually copying the libraries from
libs/linux-gcc-<ver>/*.{a,dylib} into /opt/local/lib but when I run
otool on the library I get the following :


otool -L /opt/local/lib/libjson_linux-gcc-4.0.1_libmt.dylib
/opt/local/lib/libjson_linux-gcc-4.0.1_libmt.dylib:
	buildscons/linux-gcc-4.0.1/src/lib_json/libjson_linux-gcc-4.0.1_libmt.dylib
(compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 111.1.6)
	/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0)
	/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

Also, I tried linking my test program against it and of course it
failed at runtime since it cannot find the library.

Is there a documented way of installing the library on a Mac ?

FWIW, I have tried the same with the libraries located under
buildscons/linux-gcc ... with the same results.

Help ?

Thanks,
Radu

Oops, I removed the list's mailing address by mistake. Resending to the original list.

From: P, Chandrasekhar
Sent: Wednesday, March 23, 2011 3:15 PM
To: 'Amit Rao'
Subject: RE: [Jsoncpp-devel] Potential memory leaks in jsoncpp

Hi Amit,
                Ran few valgrind tests and found no memory leaks reported. Can u attach the sample code that is reporting this leak?

Thanks,
Sekhar.


From: P, Chandrasekhar
Sent: Monday, March 21, 2011 11:13 AM
To: 'Amit Rao'
Subject: RE: [Jsoncpp-devel] Potential memory leaks in jsoncpp

Hi Amit,
                Thanks for reporting this. So far, I'm not aware of any memory leaks in JSONCPP. However, from the stack trace you've provided below it looks Json::Value copy constructor is being invoked. A review of the code looks like a valid issue to me. In the code snippet below, value_.string_ should have been freed before being assigned.

435      if ( other.value_.string_ )
436      {
437         value_.string_ = valueAllocator()->duplicateStringValue( other.value_.string_ );
438         allocated_ = true;
439      }
440      else
441         value_.string_ = 0;

Thanks,
Sekhar.

From: Amit Rao [mailto:am...@ya...]
Sent: Tuesday, March 08, 2011 6:17 AM
To: jso...@li...
Cc: Amit Rao
Subject: [Jsoncpp-devel] Potential memory leaks in jsoncpp

Hi jsoncpp-devel,

I am developing application code that uses jsoncpp-0.5.0 on a  freebsd6 server compiled with gcc version 2.95.3 20010315 (release). I am seeing that the heap memory of the process is constantly growing when a large json stucture with several thousands of entries is parsed.  The process grows to over 500 MB in size and eventually dumps core with a seg fault. Memory leak detection tools are reporting a memory leak with the following stack trace. The application code's entry point to the jsoncpp library is through the Json::Reader::parse() call. It seems that the underlying std::map under Value::Value is not deleted properly in this case.

|   allocation size per-call (min max avg): 4 94 27
|   most recent stack trace:
#0  0x213caea5 in malloc () from /usr/lib32/compat/libc.so.4
#1  0x234810a3 in Json::DefaultValueAllocator::duplicateStringValue (
    this=0x23491f9c, value=0x12c4eb0 "address", length=4294967295)
    at json_value.cpp:85
#2  0x23479917 in Json::Value::Value (this=0xffffb6a0, other=@0xffffb6ec)
    at json_value.cpp:437
#3  0x23479d73 in Json::Value::operator= (this=0x11d6130, other=@0xffffb6ec)
    at json_value.cpp:510
#4  0x23470484 in Json::Reader::decodeString (this=0x113f53c,
    token=@0xffffb744) at json_reader.cpp:652
#5  0x2346dbee in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:220
#6  0x2346ec97 in Json::Reader::readObject (this=0x113f53c,
    tokenStart=@0xffffb814) at json_reader.cpp:510
#7  0x2346dbc1 in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:211
#8  0x2346f2c7 in Json::Reader::readArray (this=0x113f53c,
    tokenStart=@0xffffb8b4) at json_reader.cpp:554
#9  0x2346dbd0 in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:214
#10 0x2346ec97 in Json::Reader::readObject (this=0x113f53c,
    tokenStart=@0xffffb984) at json_reader.cpp:510
#11 0x2346dbc1 in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:211
#12 0x2346f2c7 in Json::Reader::readArray (this=0x113f53c,
    tokenStart=@0xffffba24) at json_reader.cpp:554
#13 0x2346dbd0 in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:214
#14 0x2346ec97 in Json::Reader::readObject (this=0x113f53c,
    tokenStart=@0xffffbaf4) at json_reader.cpp:510
#15 0x2346dbc1 in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:211
#16 0x2346ec97 in Json::Reader::readObject (this=0x113f53c,
    tokenStart=@0xffffbbc4) at json_reader.cpp:510
#17 0x2346dbc1 in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:211
#18 0x2346d8d6 in Json::Reader::parse (this=0x113f53c,
    beginDoc=0x14fc010 "{\"contacts\":{\"start\":0,\"count\":618,\"total\":618,\"uri\":\"http://social.yahooapis.com/v1/user/3N74ASGDFNOAJUBXOFSDDZZEYQ/contacts\",\"contact\":[{\"uri\":\"http://social.yahooapis.com/v1/user/3N74ASGDFNOAJUBXOF"..., endDoc=0x1599ebd "", root=@0xffffbdf0, collectComments=true)
    at json_reader.cpp:172
#19 0x2346d696 in Json::Reader::parse (this=0x113f53c, document=@0xffffbd1c,
    root=@0xffffbdf0, collectComments=true) at json_reader.cpp:129
#20 0x231cff6e in ymmaddbk::AddressBookClient::parseContactsInJson (
    this=0x113f500, text=@0xffffbd1c, root=@0xffffbdf0)
    at AddressBookClient.cc:582
#21 0x231c3a21 in ymmaddbk::AddressBookClient::getAllRecords (this=0x113f500,
    ytcookie=@0x10f7688, root=@0xffffbdf0) at AddressBookClient.cc:144
#22 0x231c70c2 in ymmaddbk::AddressBookClient::getNickNames (this=0x113f500,
    ytcookie=@0x10f7688, nickNames=@0xffffbf4c) at AddressBookClient.cc:313
#23 0x231e4df3 in ymmaddbk::ymmab::getNickNames (this=0xffffbf58,
    ytcookie=@0x10f7688, nickNames=@0xffffbf4c) at ymmab.cc:347
#24 0x22c8dac0 in push::pushNotify::valAndInitialize (this=0x10f7680,
    rawData=@0xffffc060, len=@0xffffc04c) at PushNotifier.cc:384
#25 0x22c8a696 in push::pushNotifyWrapper::valAndInitialize (this=0x113e220,
    rawData=@0xffffc060, len=@0xffffc04c) at PushNotifier.cc:182
#26 0x22c9425a in MsgrPushNotifyProcess (
    consumer=0x1064838 "MsgrPushNotify.so /home/y/conf/ymmpush/pushConsumerApns.xml", entry=0x1048d10) at PushNotifier.cc:653
#27 0x2114fca2 in yreplLogConsSO::callProcess (this=0x10dd3c0,
    consumer=0x1064838 "MsgrPushNotify.so /home/y/conf/ymmpush/pushConsumerApns.xml", entry=0x1048d10) at logConsSO.cc:135
#28 0x01023212 in processEntry (entry=0x1048cd0) at txLogReader.cc:1533
#29 0x010195a5 in flushEntryBuffer (unreadLast=false) at txLogReader.cc:335
#30 0x0101a271 in addEntryToBuffer (entry=0x11790a0) at txLogReader.cc:512
#31 0x0101d9db in main (argc=2, argv=0xffffc330) at txLogReader.cc:10

Is this a known issue?  Has anyone encountered memory leaks with jsoncpp-0.5.0? If someone on the mailing list
Has run into this problem please let me know.

Thanks,
amit

Here is some additional data about the server...
FreeBSD cache101a.a1.rest.msg.sp1.yahoo.com 6.3-YAHOO-20080603 FreeBSD 6.3-YAHOO-20080603 #0: Wed Jun  4 08:19:43 PDT 2008        jh...@fb...:/home/src/sys/amd64/compile/YAHOO amd64

Hi jsoncpp-devel,

I am developing application code that uses jsoncpp-0.5.0 on a  freebsd6 server compiled with gcc version 2.95.3 20010315 (release). I am seeing that the heap memory of the process is constantly growing when a large json stucture with several thousands of entries is parsed.  The process grows to over 500 MB in size and eventually dumps core with a seg fault. Memory leak detection tools are reporting a memory leak with the following stack trace. The application code's entry point to the jsoncpp library is through the Json::Reader::parse() call. It seems that the underlying std::map under Value::Value is not deleted properly in this case.

|   allocation size per-call (min max avg): 4 94 27
|   most recent stack trace:
#0  0x213caea5 in malloc () from /usr/lib32/compat/libc.so.4
#1  0x234810a3 in Json::DefaultValueAllocator::duplicateStringValue (
    this=0x23491f9c, value=0x12c4eb0 "address", length=4294967295)
    at json_value.cpp:85
#2  0x23479917 in Json::Value::Value (this=0xffffb6a0, other=@0xffffb6ec)
    at json_value.cpp:437
#3  0x23479d73 in Json::Value::operator= (this=0x11d6130, other=@0xffffb6ec)
    at json_value.cpp:510
#4  0x23470484 in Json::Reader::decodeString (this=0x113f53c,
    token=@0xffffb744) at json_reader.cpp:652
#5  0x2346dbee in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:220
#6  0x2346ec97 in Json::Reader::readObject (this=0x113f53c,
    tokenStart=@0xffffb814) at json_reader.cpp:510
#7  0x2346dbc1 in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:211
#8  0x2346f2c7 in Json::Reader::readArray (this=0x113f53c,
    tokenStart=@0xffffb8b4) at json_reader.cpp:554
#9  0x2346dbd0 in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:214
#10 0x2346ec97 in Json::Reader::readObject (this=0x113f53c,
    tokenStart=@0xffffb984) at json_reader.cpp:510
#11 0x2346dbc1 in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:211
#12 0x2346f2c7 in Json::Reader::readArray (this=0x113f53c,
    tokenStart=@0xffffba24) at json_reader.cpp:554
#13 0x2346dbd0 in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:214
#14 0x2346ec97 in Json::Reader::readObject (this=0x113f53c,
    tokenStart=@0xffffbaf4) at json_reader.cpp:510
#15 0x2346dbc1 in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:211
#16 0x2346ec97 in Json::Reader::readObject (this=0x113f53c,
    tokenStart=@0xffffbbc4) at json_reader.cpp:510
#17 0x2346dbc1 in Json::Reader::readValue (this=0x113f53c)
    at json_reader.cpp:211
#18 0x2346d8d6 in Json::Reader::parse (this=0x113f53c,
    beginDoc=0x14fc010 "{\"contacts\":{\"start\":0,\"count\":618,\"total\":618,\"uri\":\"http://social.yahooapis.com/v1/user/3N74ASGDFNOAJUBXOFSDDZZEYQ/contacts\",\"contact\":[{\"uri\":\"http://social.yahooapis.com/v1/user/3N74ASGDFNOAJUBXOF"..., endDoc=0x1599ebd "", root=@0xffffbdf0, collectComments=true)
    at json_reader.cpp:172
#19 0x2346d696 in Json::Reader::parse (this=0x113f53c, document=@0xffffbd1c,
    root=@0xffffbdf0, collectComments=true) at json_reader.cpp:129
#20 0x231cff6e in ymmaddbk::AddressBookClient::parseContactsInJson (
    this=0x113f500, text=@0xffffbd1c, root=@0xffffbdf0)
    at AddressBookClient.cc:582
#21 0x231c3a21 in ymmaddbk::AddressBookClient::getAllRecords (this=0x113f500,
    ytcookie=@0x10f7688, root=@0xffffbdf0) at AddressBookClient.cc:144
#22 0x231c70c2 in ymmaddbk::AddressBookClient::getNickNames (this=0x113f500,
    ytcookie=@0x10f7688, nickNames=@0xffffbf4c) at AddressBookClient.cc:313
#23 0x231e4df3 in ymmaddbk::ymmab::getNickNames (this=0xffffbf58,
    ytcookie=@0x10f7688, nickNames=@0xffffbf4c) at ymmab.cc:347
#24 0x22c8dac0 in push::pushNotify::valAndInitialize (this=0x10f7680,
    rawData=@0xffffc060, len=@0xffffc04c) at PushNotifier.cc:384
#25 0x22c8a696 in push::pushNotifyWrapper::valAndInitialize (this=0x113e220,
    rawData=@0xffffc060, len=@0xffffc04c) at PushNotifier.cc:182
#26 0x22c9425a in MsgrPushNotifyProcess (
    consumer=0x1064838 "MsgrPushNotify.so /home/y/conf/ymmpush/pushConsumerApns.xml", entry=0x1048d10) at PushNotifier.cc:653
#27 0x2114fca2 in yreplLogConsSO::callProcess (this=0x10dd3c0,
    consumer=0x1064838 "MsgrPushNotify.so /home/y/conf/ymmpush/pushConsumerApns.xml", entry=0x1048d10) at logConsSO.cc:135
#28 0x01023212 in processEntry (entry=0x1048cd0) at txLogReader.cc:1533
#29 0x010195a5 in flushEntryBuffer (unreadLast=false) at txLogReader.cc:335
#30 0x0101a271 in addEntryToBuffer (entry=0x11790a0) at txLogReader.cc:512
#31 0x0101d9db in main (argc=2, argv=0xffffc330) at txLogReader.cc:10

Is this a known issue?  Has anyone encountered memory leaks with jsoncpp-0.5.0? If someone on the mailing list
Has run into this problem please let me know.

Thanks,
amit

Here is some additional data about the server...
FreeBSD cache101a.a1.rest.msg.sp1.yahoo.com 6.3-YAHOO-20080603 FreeBSD 6.3-YAHOO-20080603 #0: Wed Jun  4 08:19:43 PDT 2008        jh...@fb...:/home/src/sys/amd64/compile/YAHOO amd64

Hello,
	Is the below patch applied to SVN?
Thanks,
Sekhar.

-----Original Message-----
From: P, Chandrasekhar 
Sent: Monday, January 31, 2011 12:53 PM
To: 'jso...@li...'
Subject: Patch for Bug ID: 3062498(Bug when parsing duplicate names)

Hello Baptiste,
	
Pls review and apply the following patches into SVN as a bug fix:

*** jsoncpp-src-0.5.0/include/json/features.h   2010-03-12 13:01:35.000000000 +0530
--- jsoncpp-src-0.5.0-latest/include/json/features.h    2011-01-11 16:08:15.000000000 +0530
***************
*** 23,28 ****
--- 23,29 ----
         * - Comments are forbidden.
         * - Root object must be either an array or an object value.
         * - Assumes Value strings are encoded in UTF-8
+        * - Duplicate keys are NOT allowed within objects.
         */
        static Features strictMode();

***************
*** 35,40 ****
--- 36,46 ----

        /// \c true if root must be either an array or an object value. Default: \c false.
        bool strictRoot_;
+
+       /** \c true if only unique keys are allowed within objects. Default: \c false.
+         * - RFC4627 mentions that key name SHOULD be unique
+         */
+       bool uniqueKeys_;
     };

  } // namespace Json

*** jsoncpp-src-0.5.0/src/lib_json/json_reader.cpp      2010-03-12 13:01:35.000000000 +0530
--- jsoncpp-src-0.5.0-latest/src/lib_json/json_reader.cpp       2011-01-11 16:50:45.000000000 +0530
***************
*** 19,24 ****
--- 19,25 ----
  Features::Features()
     : allowComments_( true )
     , strictRoot_( false )
+    , uniqueKeys_(false)
  {
  }

***************
*** 36,41 ****
--- 37,43 ----
     Features features;
     features.allowComments_ = false;
     features.strictRoot_ = true;
+    features.uniqueKeys_ = true;
     return features;
  }

***************
*** 505,510 ****
--- 507,518 ----
                                      colon,
                                      tokenObjectEnd );
        }
+         if(features_.uniqueKeys_ && currentValue().isMember(name))
+         {
+               return addErrorAndRecover( "Duplicate key found",
+                                   tokenName,
+                                   tokenObjectEnd );
+         }
        Value &value = currentValue()[ name ];
        nodes_.push( &value );
        bool ok = readValue();

thanks,
sekhar.

Hello Baptiste,
	
Pls review and apply the following patches into SVN as a bug fix:

*** jsoncpp-src-0.5.0/include/json/features.h   2010-03-12 13:01:35.000000000 +0530
--- jsoncpp-src-0.5.0-latest/include/json/features.h    2011-01-11 16:08:15.000000000 +0530
***************
*** 23,28 ****
--- 23,29 ----
         * - Comments are forbidden.
         * - Root object must be either an array or an object value.
         * - Assumes Value strings are encoded in UTF-8
+        * - Duplicate keys are NOT allowed within objects.
         */
        static Features strictMode();

***************
*** 35,40 ****
--- 36,46 ----

        /// \c true if root must be either an array or an object value. Default: \c false.
        bool strictRoot_;
+
+       /** \c true if only unique keys are allowed within objects. Default: \c false.
+         * - RFC4627 mentions that key name SHOULD be unique
+         */
+       bool uniqueKeys_;
     };

  } // namespace Json

*** jsoncpp-src-0.5.0/src/lib_json/json_reader.cpp      2010-03-12 13:01:35.000000000 +0530
--- jsoncpp-src-0.5.0-latest/src/lib_json/json_reader.cpp       2011-01-11 16:50:45.000000000 +0530
***************
*** 19,24 ****
--- 19,25 ----
  Features::Features()
     : allowComments_( true )
     , strictRoot_( false )
+    , uniqueKeys_(false)
  {
  }

***************
*** 36,41 ****
--- 37,43 ----
     Features features;
     features.allowComments_ = false;
     features.strictRoot_ = true;
+    features.uniqueKeys_ = true;
     return features;
  }

***************
*** 505,510 ****
--- 507,518 ----
                                      colon,
                                      tokenObjectEnd );
        }
+         if(features_.uniqueKeys_ && currentValue().isMember(name))
+         {
+               return addErrorAndRecover( "Duplicate key found",
+                                   tokenName,
+                                   tokenObjectEnd );
+         }
        Value &value = currentValue()[ name ];
        nodes_.push( &value );
        bool ok = readValue();

thanks,
sekhar.

Hi Baptiste,

Thank you for your reply. 

Sorry for this confusion. To elaborate my point (use case), I want to write/read 
each 4K as a separate JSON structure/object that is enclosed within {}. For 
example, each 4K JSON object would be stored as one entity/blob. This entity 
will need to be a composite array (multiple arrays) for my use case enclosing 
it with [] i.e. ["value1", "value2", "value3"].

I was thinking to store each 4K blob in an in-memory db in JSON for easier 
composition/decomposition of the data, but this JSON takes a lot of extra space 
due to its inherent tags like {}, [], which I referred as a separator. For 1M 
records, the total memory size for my use case is then 4GB. However, my use case 
could go even higher, e.g. 64G. So, I was also looking into the JsonCpp library 
to see if there is any data compression capability added to it, but could not 
find anything. 


You are probably right that JSON is probably not a good choice for such a large 
in-memory storage. Any further thoughts?

Thanks,
Anath B. Mandal




________________________________
From: Baptiste Lepilleur <bap...@gm...>
To: Anath Bandhu Mandal <a_b...@ya...>
Cc: jsoncpp-devel <jso...@li...>; blep 
<bl...@us...>
Sent: Thu, 16 September, 2010 13:49:41
Subject: Re: Example code for Jsoncpp library


2010/9/15 Anath Bandhu Mandal <a_b...@ya...>

Hi
>
>I have been using the jsoncpp library, but having difficulties to understand how 
>to effectively read/write data using Json::Value objects. You have give one 
>example on http://jsoncpp.sourceforge.net/index.html, which does not explore 
>much of the classes on how are they should be used. For example, I want to store 
>about 4K of data with separator in one array, but this array should be added 
>for a million records for read/write. However, I don't find an effective to way 
>utilize this library due to limited example code.
>
>Could you please provide some example code on how to effective utilize the 
>member functions for Json::Value, Json::Reader, and Json::Writer please?

You can look at the code of the jsontestrunner that is used for testing 
reader/writer (e.g. it reads a json file, and rewrite it in json, as well as a 
custom format).

That being said, I don't think jsoncpp is a good choice of library for your use 
case: if I understood correctly (not sure what you mean by data with separator), 
you want to parse 4K * 1M values. Because the parser will load the whole json 
file in memory, a lot of memory is going to be wasted (I'm not even sure you 
will be able to fit the Json::Value tree in memory after conversion: 
4K*1M=4*10**9 => means about 4GB of memory required if each value needed only 1 
byte!). Writing should not be an issue as you can use the stream based writer 
that allow incremental writing to file. 

Baptiste.


>
>Thanks,
>AB Mandal
>



      

2010/9/15 Anath Bandhu Mandal <a_b...@ya...>

> Hi
>
> I have been using the jsoncpp library, but having difficulties to
> understand how to effectively read/write data using Json::Value objects. You
> have give one example on http://jsoncpp.sourceforge.net/index.html, which
> does not explore much of the classes on how are they should be used. For
> example, I want to store about 4K of data with separator in one array,
> but this array should be added for a million records for read/write.
> However, I don't find an effective to way utilize this library due to
> limited example code.
>
> Could you please provide some example code on how to effective utilize the
> member functions for Json::Value, Json::Reader, and Json::Writer please?
>

You can look at the code of the jsontestrunner that is used for testing
reader/writer (e.g. it reads a json file, and rewrite it in json, as well as
a custom format).

That being said, I don't think jsoncpp is a good choice of library for your
use case: if I understood correctly (not sure what you mean by data with
separator), you want to parse 4K * 1M values. Because the parser will load
the whole json file in memory, a lot of memory is going to be wasted (I'm
not even sure you will be able to fit the Json::Value tree in memory after
conversion: 4K*1M=4*10**9 => means about 4GB of memory required if each
value needed only 1 byte!). Writing should not be an issue as you can use
the stream based writer that allow incremental writing to file.

Baptiste.


> Thanks,
> AB Mandal
>
>

Hi

I have been using the jsoncpp library, but having difficulties to understand how 
to effectively read/write data using Json::Value objects. You have give one 
example on http://jsoncpp.sourceforge.net/index.html, which does not explore 
much of the classes on how are they should be used. For example, I want to store 
about 4K of data with separator in one array, but this array should be added 
for a million records for read/write. However, I don't find an effective to way 
utilize this library due to limited example code.

Could you please provide some example code on how to effective utilize the 
member functions for Json::Value, Json::Reader, and Json::Writer please?

Thanks,
AB Mandal


      

2010/8/30 Christian Auby <chr...@au...>

> If I create a Json::Value like this:
>
> Json::Value v;
>
> v["b"] = 5;
> v["c"] = 6;
> v["a"] = 7;
>
> The order in the created JSON output string will be a, b and c. Is it
> possible to have them appear in the order they were first assigned (created
> if you will)? This would aid a lot in debugging as the values will appear in
> the order they are in the code. I also seem to have read in one of the
> specifications that in one specific case the order was significant, but I
> can't for the life of me find it again.
>

I don't remember ever seeing this features. The other JSON  implementation I
know, such as python json module, use an unordered map to store the
key/value association. At best, you can usually hint that you would like key
to be sorted alphabetically when encoding the data in JSON.

Baptiste.


> Please CC me on reply.
>
> --
> Christian
>
>

If I create a Json::Value like this:

Json::Value v;

v["b"] = 5;
v["c"] = 6;
v["a"] = 7;

The order in the created JSON output string will be a, b and c. Is it
possible to have them appear in the order they were first assigned (created
if you will)? This would aid a lot in debugging as the values will appear in
the order they are in the code. I also seem to have read in one of the
specifications that in one specific case the order was significant, but I
can't for the life of me find it again.

Please CC me on reply.

-- 
Christian

Hi,

I am new to JSON, and trying to implement a writer.
The problem is unicode characters.

The function valueToQuotedString has a comment:

std::string valueToQuotedString( const char *value )
{
   // Not sure how to handle unicode...
.
..
}

So if we have a sample string like this: 'Niittylä', it gets transformed into : 'Niittylä'.

Please advise how to provide support for unicode.

Regards,
/SK

2010/4/21 <ma...@al...>

>  [...]
>
> > Should we add asWString() and asCWString() ? If so should they return the
> string encoded in utf-16 or utf-32 on platforms where wchar_t is 32 bits?
> The string that was passed at initialization? Or should we makes encoding
> explicit: asUTF16WString(), asUTF32WString()? Concerning the last options,
> as UTF32 would not be possible on platforms where wchar_t is 2 bytes, this
> would introduce portability issues...
> Yep, better to have asWString() and asCWString(), and they always should
> return string encoded in utf-16.
>
> What is your rational for always returning string encoded in utf-16? I'm
afraid that if we do so on a system where wchar_t is encoded using utf-32,
the resulting string would not be compatible with system w* API, such as
towlower... listed in part here:
http://www.unix.org/version2/whatsnew/login_mse.html.

On Tue, Apr 20, 2010 at 5:25 PM, Baptiste Lepilleur <
bap...@gm...> wrote:

> 2010/4/19 <ma...@al...>
>
>> The application uses std:wstring for JSON message values due to unicode
>> maintenance,
>>
>
Sorry, one more point: as far as i'm aware, JSON uses only utf8, not utf16
(though most JS interpreters support utf16 script input and/or use utf16
internally as their native string type).

If that is indeed true, i believe that utf16 support in jsoncpp is a bit
superfluous, except to allow clients to import/export their utf16 data into
jsoncpp's internal format (ascii/utf8?).

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/

On Tue, Apr 20, 2010 at 5:39 PM, Stephan Beal <sg...@go...> wrote:

> is VERY easy to use and is Public Domain. It can convert/very utf8/16/32
> and has a very handy
>

convert/very == convert/verIFy

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/

On Tue, Apr 20, 2010 at 5:25 PM, Baptiste Lepilleur <
bap...@gm...> wrote:

> Unicode awarness in JsonCpp is fairly recent (e.g. handling of unicode
> escape sequence). And IMHO we still need more tests (such as testing if we
> correctly handle surrogate escape sequences).
>

If i may recommend:

http://utfcpp.sourceforge.net/

is VERY easy to use and is Public Domain. It can convert/very utf8/16/32 and
has a very handy iterator class which lets you iterator over utf8/16/32
strings in a sane manner (each iteration returns on logical character,
regardless of its real length).


> implementation: typically 2 bytes on MSVC, and 4 bytes with gcc). Side
> note: the next C++ standard introduces utf16_t and utf32_t types to make
> this explicit,
>

Yeah!!!!


> but those are not yet widely available in the industry, so I'd rather we do
> not rely on this yet.
>

:(


> - What asString() and asCString() should return when initialized with an
> std::wstring? The string converted in utf-8?
>

utfcpp makes the conversion to utf8 trivial:

utf16to8( inputIteratorBegin, inputIteratorEnd, outputIterator)

e.g., something like:

std::string u8;
utf16to8( wstr.begin(), wstr.end(), std::back_inserter( u8 ) );

i only recently started using utfcpp, but i'm very impressed with how easy
it is to use (i'm no Unicode expert, so i need tools like this to help me
:).

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/

2010/4/19 <ma...@al...>

>  Hello Baptiste !
>
>
>
> I decided to use jsoncpp in my application, but faced an issue !
>
> The application uses std:wstring for JSON message values due to unicode
> maintenance,
>
> but I don’t see how I can form JSON message through Json::Value.
>
> So, why it doesn’t support std::wstring and how it could be solved ?
>
Unicode awarness in JsonCpp is fairly recent (e.g. handling of unicode
escape sequence). And IMHO we still need more tests (such as testing if we
correctly handle surrogate escape sequences).

You are the first to ask this features. Though, on Windows it is fairly
common to represent utf_16 encoded string as wstring, as it is the native
encoding of the O.S.. But as Stephan pointed out, the C++ standard specify
neither the size of wchar_t (and reality show varying implementation:
typically 2 bytes on MSVC, and 4 bytes with gcc). Side note: the next C++
standard introduces utf16_t and utf32_t types to make this explicit, but
those are not yet widely available in the industry, so I'd rather we do not
rely on this yet.

We could fairly easily adds support to set a Json::Value from a
std::wstring, since utf-16 and utf-32 encoding are not conflicting (please
correct me if I'm wrong).  This raises a few questions:
- should we store the wstring as a sequence of char after conversion to
utf-8, or should it be stored "as this".
- What asString() and asCString() should return when initialized with an
std::wstring? The string converted in utf-8?
- Should we add asWString() and asCWString() ? If so should they return the
string encoded in utf-16 or utf-32 on platforms where wchar_t is 32 bits?
The string that was passed at initialization? Or should we makes encoding
explicit: asUTF16WString(), asUTF32WString()? Concerning the last options,
as UTF32 would not be possible on platforms where wchar_t is 2 bytes, this
would introduce portability issues...

Baptiste.


>
>
> Regards.
>
> * *
>
> *Maxim Aga*
>

On Mon, Apr 19, 2010 at 4:37 PM, <ma...@al...> wrote:

>  I decided to use jsoncpp in my application, but faced an issue !
>
> The application uses std:wstring for JSON message values due to unicode
> maintenance,
>
> but I don’t see how I can form JSON message through Json::Value.
>
> So, why it doesn’t support std::wstring and how it could be solved ?
>

std::wstring uses an unspecified character size and (AFAIK) byte ordering.
e.g. gcc's wstring uses 4-byte characters. i recently implemented a
conversion from wstring to std::basic_string<uint16_t> for use with utf8cpp,
and it looks like this:

ValueType == basic_string<uint16_t>

    void Utf16String::init( wchar_t const * v, size_t len )
    {
        size_t const sl = (v&&len) ? len : 0;
        if( ! sl )
        {
            this->init( ValueType(), 0 );
            return;
        }
        ValueType vec;
        vec.reserve( sl );
        vec.assign( v, v+sl );
        this->init( vec, 0);
    }

    Utf16String::Utf16String( std::wstring const & v )
    {
        this->init( v.empty() ? 0 : v.c_str(), v.size() );
    }

however, i don't know if that will work as-is for both big- and
little-endian.

i hope this helps a little bit.

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/

Hello Baptiste !

I decided to use jsoncpp in my application, but faced an issue !
The application uses std:wstring for JSON message values due to unicode maintenance,
but I don't see how I can form JSON message through Json::Value.
So, why it doesn't support std::wstring and how it could be solved ?

Regards.

Maxim Aga

Hi all,
It has recently come to my attention that public domain is not valid in all
countries (Germany for example).

To solve this, I plan to follow Stephan Beal stance on this:
http://code.google.com/p/whefs/source/browse/trunk/LICENSE

Basically provides MIT license, and optionally public domain when desired
and applicable.

Does anyone forsee any problems with this change?

Baptiste.