jsoncpp-devel Mailing List for json-cpp

2010/4/21 <ma...@al...>

>  [...]
>
> > Should we add asWString() and asCWString() ? If so should they return the
> string encoded in utf-16 or utf-32 on platforms where wchar_t is 32 bits?
> The string that was passed at initialization? Or should we makes encoding
> explicit: asUTF16WString(), asUTF32WString()? Concerning the last options,
> as UTF32 would not be possible on platforms where wchar_t is 2 bytes, this
> would introduce portability issues...
> Yep, better to have asWString() and asCWString(), and they always should
> return string encoded in utf-16.
>
> What is your rational for always returning string encoded in utf-16? I'm
afraid that if we do so on a system where wchar_t is encoded using utf-32,
the resulting string would not be compatible with system w* API, such as
towlower... listed in part here:
http://www.unix.org/version2/whatsnew/login_mse.html.

On Tue, Apr 20, 2010 at 5:25 PM, Baptiste Lepilleur <
bap...@gm...> wrote:

> 2010/4/19 <ma...@al...>
>
>> The application uses std:wstring for JSON message values due to unicode
>> maintenance,
>>
>
Sorry, one more point: as far as i'm aware, JSON uses only utf8, not utf16
(though most JS interpreters support utf16 script input and/or use utf16
internally as their native string type).

If that is indeed true, i believe that utf16 support in jsoncpp is a bit
superfluous, except to allow clients to import/export their utf16 data into
jsoncpp's internal format (ascii/utf8?).

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/

On Tue, Apr 20, 2010 at 5:39 PM, Stephan Beal <sg...@go...> wrote:

> is VERY easy to use and is Public Domain. It can convert/very utf8/16/32
> and has a very handy
>

convert/very == convert/verIFy

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/

On Tue, Apr 20, 2010 at 5:25 PM, Baptiste Lepilleur <
bap...@gm...> wrote:

> Unicode awarness in JsonCpp is fairly recent (e.g. handling of unicode
> escape sequence). And IMHO we still need more tests (such as testing if we
> correctly handle surrogate escape sequences).
>

If i may recommend:

http://utfcpp.sourceforge.net/

is VERY easy to use and is Public Domain. It can convert/very utf8/16/32 and
has a very handy iterator class which lets you iterator over utf8/16/32
strings in a sane manner (each iteration returns on logical character,
regardless of its real length).


> implementation: typically 2 bytes on MSVC, and 4 bytes with gcc). Side
> note: the next C++ standard introduces utf16_t and utf32_t types to make
> this explicit,
>

Yeah!!!!


> but those are not yet widely available in the industry, so I'd rather we do
> not rely on this yet.
>

:(


> - What asString() and asCString() should return when initialized with an
> std::wstring? The string converted in utf-8?
>

utfcpp makes the conversion to utf8 trivial:

utf16to8( inputIteratorBegin, inputIteratorEnd, outputIterator)

e.g., something like:

std::string u8;
utf16to8( wstr.begin(), wstr.end(), std::back_inserter( u8 ) );

i only recently started using utfcpp, but i'm very impressed with how easy
it is to use (i'm no Unicode expert, so i need tools like this to help me
:).

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/

2010/4/19 <ma...@al...>

>  Hello Baptiste !
>
>
>
> I decided to use jsoncpp in my application, but faced an issue !
>
> The application uses std:wstring for JSON message values due to unicode
> maintenance,
>
> but I don’t see how I can form JSON message through Json::Value.
>
> So, why it doesn’t support std::wstring and how it could be solved ?
>
Unicode awarness in JsonCpp is fairly recent (e.g. handling of unicode
escape sequence). And IMHO we still need more tests (such as testing if we
correctly handle surrogate escape sequences).

You are the first to ask this features. Though, on Windows it is fairly
common to represent utf_16 encoded string as wstring, as it is the native
encoding of the O.S.. But as Stephan pointed out, the C++ standard specify
neither the size of wchar_t (and reality show varying implementation:
typically 2 bytes on MSVC, and 4 bytes with gcc). Side note: the next C++
standard introduces utf16_t and utf32_t types to make this explicit, but
those are not yet widely available in the industry, so I'd rather we do not
rely on this yet.

We could fairly easily adds support to set a Json::Value from a
std::wstring, since utf-16 and utf-32 encoding are not conflicting (please
correct me if I'm wrong).  This raises a few questions:
- should we store the wstring as a sequence of char after conversion to
utf-8, or should it be stored "as this".
- What asString() and asCString() should return when initialized with an
std::wstring? The string converted in utf-8?
- Should we add asWString() and asCWString() ? If so should they return the
string encoded in utf-16 or utf-32 on platforms where wchar_t is 32 bits?
The string that was passed at initialization? Or should we makes encoding
explicit: asUTF16WString(), asUTF32WString()? Concerning the last options,
as UTF32 would not be possible on platforms where wchar_t is 2 bytes, this
would introduce portability issues...

Baptiste.


>
>
> Regards.
>
> * *
>
> *Maxim Aga*
>

On Mon, Apr 19, 2010 at 4:37 PM, <ma...@al...> wrote:

>  I decided to use jsoncpp in my application, but faced an issue !
>
> The application uses std:wstring for JSON message values due to unicode
> maintenance,
>
> but I don’t see how I can form JSON message through Json::Value.
>
> So, why it doesn’t support std::wstring and how it could be solved ?
>

std::wstring uses an unspecified character size and (AFAIK) byte ordering.
e.g. gcc's wstring uses 4-byte characters. i recently implemented a
conversion from wstring to std::basic_string<uint16_t> for use with utf8cpp,
and it looks like this:

ValueType == basic_string<uint16_t>

    void Utf16String::init( wchar_t const * v, size_t len )
    {
        size_t const sl = (v&&len) ? len : 0;
        if( ! sl )
        {
            this->init( ValueType(), 0 );
            return;
        }
        ValueType vec;
        vec.reserve( sl );
        vec.assign( v, v+sl );
        this->init( vec, 0);
    }

    Utf16String::Utf16String( std::wstring const & v )
    {
        this->init( v.empty() ? 0 : v.c_str(), v.size() );
    }

however, i don't know if that will work as-is for both big- and
little-endian.

i hope this helps a little bit.

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/

Hello Baptiste !

I decided to use jsoncpp in my application, but faced an issue !
The application uses std:wstring for JSON message values due to unicode maintenance,
but I don't see how I can form JSON message through Json::Value.
So, why it doesn't support std::wstring and how it could be solved ?

Regards.

Maxim Aga

Hi all,
It has recently come to my attention that public domain is not valid in all
countries (Germany for example).

To solve this, I plan to follow Stephan Beal stance on this:
http://code.google.com/p/whefs/source/browse/trunk/LICENSE

Basically provides MIT license, and optionally public domain when desired
and applicable.

Does anyone forsee any problems with this change?

Baptiste.

My appologies if this hits the list twice... the jsoncpp docs say to send
comments to the list but don't say it is a subscriber-only list....

---------- Forwarded message ----------
From: Stephan Beal <sg...@go...>
Date: Thu, Apr 8, 2010 at 4:41 PM
Subject: Fwd: jsoncpp and libs11n
To: jso...@li...


i finally found the dev list address at the bottom/right of the doxygen
docs, so i'm re-sending this to the list...


---------- Forwarded message ----------
From: Stephan Beal <sg...@go...>
Date: Thu, Apr 8, 2010 at 3:17 PM
Subject: jsoncpp and libs11n
To: bl...@us...


Hi, Babtiste!

i am the author of the libs11n project (http://s11n.net), a generic
serialization framework for C++, and i wanted to take a moment to tell you
how i'm using jsoncpp...

Brief overview: s11n is, unlike most serialization frameworks, ignorant of
any data types. Like jsoncpp, it uses a DOM-like structure, and arbitrary
i/o formatters can be written which use those DOM-like structures. s11n
currently has 6 or 8 different data formats, and was the first serialization
library ever (AFAIK) to be able to serialize C++ objects directly to email (
http://blog.s11n.net/?p=17).

Anyway...

i only came across jsoncpp yesterday, during my semi-monthly searches for
C/C++-based JSON i/o handlers. So far jsoncpp is the only one where i've
look at it and said, "yes! This is an API i can enjoy using!"

Last night i started writing an i/o handler based on your code (with only 1
small patch), and we can now serialize near-arbitrary C++ objects to json
:). Over the weekend i'll get the input support working. i was going to
suggest the addition of an event-based parser (it would significantly cut
the memory costs for deserialization in my case), but i see you've added it
to your jsoncpp roadmap :).

As a demonstration, here we'll serialize a std::vector< std::list<T> > ...

int main()
{
    using namespace Json;
    namespace sio = s11n::io;
    namespace sli = s11nlite;
    typedef s11n::Detail::variant Var;
    if(1)
    {
        typedef Var ValueT;
        typedef std::list<ValueT> LT;

        LT list;
        int i = 0;
        for( i = 0; i < 5; ++i )
        {
            list.push_back( "Var #"+Var(i).str() );
            list.push_back( Var(i*1.33) );
            list.push_back( Var(i) );
        }

        sli::node_type no;
        typedef sio::json_serializer<sli::node_type> SerT;
        s11nlite::serialize( no, list );
        SerT ser;
        //ser.serialize( no, std::cout );

        typedef std::vector< LT > VT;
        VT vec;
        for( i = 0; i < 2; ++i )
        {
            vec.push_back( list );
        }
        //sli::node_traits::clear( no );
        //s11nlite::serialize( no, list );
        sli::node_traits::clear( no );
        s11nlite::serialize( no, vec );
        ser.serialize( no, std::cout );
    }
    return 0;
}

that produces output like:

{"s11nIoJsonSerializer":
{"name":"s11n_node", "s11nClass":"vector",
         "children":[ {"name":"s11n_node", "s11nClass":"list",
"properties":{ "x00":"Var #0", "x01":0, "x02":0, "x03":"Var #1",
"x04":1.330, "x05":1, "x06":"Var #2", "x07":2.660, "x08":2, "x09":"Var #3",
"x0a":3.990, "x0b":3, "x0c":"Var #4", "x0d":5.320, "x0e":4 } } ]
...
}
}

(the 'x' prefixes on the properties are there for long historical reasons
involving requirements for 'key' elements for certain data formats... only
array/list-like classes do that.)

Not only that, but we can easily convert between any of s11n's supported
formats (i think it now has 7 or 8) and json by using s11nconvert (
http://s11n.net/s11nconvert/) or the C++ API (it only takes a couple lines
of code to convert arbitrary serialized objects).

i have a couple tiny additions to jsoncpp i would like to submit for your
consideration. What channel do you prefer for that? Is here a mailing list
or google group?

Anyway... i thought you might like to know that someone's getting some good
use out of your code.

Happy hacking!! And THANK YOU VERY MUCH for releasing jsoncpp into the
Public Domain (http://s11n.net/license.php).

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/



-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/



-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/

i finally found the dev list address at the bottom/right of the doxygen
docs, so i'm re-sending this to the list...

---------- Forwarded message ----------
From: Stephan Beal <sg...@go...>
Date: Thu, Apr 8, 2010 at 3:17 PM
Subject: jsoncpp and libs11n
To: bl...@us...


Hi, Babtiste!

i am the author of the libs11n project (http://s11n.net), a generic
serialization framework for C++, and i wanted to take a moment to tell you
how i'm using jsoncpp...

Brief overview: s11n is, unlike most serialization frameworks, ignorant of
any data types. Like jsoncpp, it uses a DOM-like structure, and arbitrary
i/o formatters can be written which use those DOM-like structures. s11n
currently has 6 or 8 different data formats, and was the first serialization
library ever (AFAIK) to be able to serialize C++ objects directly to email (
http://blog.s11n.net/?p=17).

Anyway...

i only came across jsoncpp yesterday, during my semi-monthly searches for
C/C++-based JSON i/o handlers. So far jsoncpp is the only one where i've
look at it and said, "yes! This is an API i can enjoy using!"

Last night i started writing an i/o handler based on your code (with only 1
small patch), and we can now serialize near-arbitrary C++ objects to json
:). Over the weekend i'll get the input support working. i was going to
suggest the addition of an event-based parser (it would significantly cut
the memory costs for deserialization in my case), but i see you've added it
to your jsoncpp roadmap :).

As a demonstration, here we'll serialize a std::vector< std::list<T> > ...

int main()
{
    using namespace Json;
    namespace sio = s11n::io;
    namespace sli = s11nlite;
    typedef s11n::Detail::variant Var;
    if(1)
    {
        typedef Var ValueT;
        typedef std::list<ValueT> LT;

        LT list;
        int i = 0;
        for( i = 0; i < 5; ++i )
        {
            list.push_back( "Var #"+Var(i).str() );
            list.push_back( Var(i*1.33) );
            list.push_back( Var(i) );
        }

        sli::node_type no;
        typedef sio::json_serializer<sli::node_type> SerT;
        s11nlite::serialize( no, list );
        SerT ser;
        //ser.serialize( no, std::cout );

        typedef std::vector< LT > VT;
        VT vec;
        for( i = 0; i < 2; ++i )
        {
            vec.push_back( list );
        }
        //sli::node_traits::clear( no );
        //s11nlite::serialize( no, list );
        sli::node_traits::clear( no );
        s11nlite::serialize( no, vec );
        ser.serialize( no, std::cout );
    }
    return 0;
}

that produces output like:

{"s11nIoJsonSerializer":
{"name":"s11n_node", "s11nClass":"vector",
         "children":[ {"name":"s11n_node", "s11nClass":"list",
"properties":{ "x00":"Var #0", "x01":0, "x02":0, "x03":"Var #1",
"x04":1.330, "x05":1, "x06":"Var #2", "x07":2.660, "x08":2, "x09":"Var #3",
"x0a":3.990, "x0b":3, "x0c":"Var #4", "x0d":5.320, "x0e":4 } } ]
...
}
}

(the 'x' prefixes on the properties are there for long historical reasons
involving requirements for 'key' elements for certain data formats... only
array/list-like classes do that.)

Not only that, but we can easily convert between any of s11n's supported
formats (i think it now has 7 or 8) and json by using s11nconvert (
http://s11n.net/s11nconvert/) or the C++ API (it only takes a couple lines
of code to convert arbitrary serialized objects).

i have a couple tiny additions to jsoncpp i would like to submit for your
consideration. What channel do you prefer for that? Is here a mailing list
or google group?

Anyway... i thought you might like to know that someone's getting some good
use out of your code.

Happy hacking!! And THANK YOU VERY MUCH for releasing jsoncpp into the
Public Domain (http://s11n.net/license.php).

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/



-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/