[Computer-go] CNN with 54% prediction on KGS 6d+ data

[Hiroshi Yamashita]

Hi Detlef,

Thank you for publishing your data and latest oakform code!
It was very helpful for me.

I tried your 54% data with Aya.

Aya with Detlef54% vs Aya with Detlef44%, 10000 playout/move
Aya with Detlef54%'s winrate is 0.569 (124wins / 218games).

CGOS BayseElo rating
Aya with Detlef44%  (aya786n_Detlef_10k) 3040
Aya with Detlef54%  (Aya786m_Det54_10k ) 3036
http://www.yss-aya.com/cgos/19x19/bayes.html

Detlef54% is a bit stronger in selfplay, but they are similar on CGOS.
Maybe Detlef54%'s prediction is strong, and Aya's playout strength
 is not enough.

Speed for a position on GTS 450.
Detlef54%   21ms
Detlef44%   17ms

Cumulative accuracy from 1000 pro games.

move rank  Aya    Detlef54%  Mixture
    1      40.8      47.6     48.0
    2      53.5      62.4     62.7
    3      60.2      70.7     71.0
    4      64.8      75.8     76.1
    5      68.1      79.5     79.9
    6      71.0      82.3     82.6
    7      73.2      84.5     84.8
    8      75.2      86.3     86.6
    9      76.9      87.8     88.1
   10      78.3      89.0     89.3
   11      79.6      90.2     90.6
   12      80.8      91.2     91.4
   13      81.9      92.0     92.2
   14      82.9      92.7     92.9
   15      83.8      93.3     93.5
   16      84.6      93.9     94.1
   17      85.4      94.3     94.5
   18      86.1      94.8     95.0
   19      86.8      95.2     95.4
   20      87.4      95.5     95.7

Mixture is pretty same as Detlef54%.
I changed learning method from MM to LFR.
Aya's own accuracy is from LFR rank, not MM gamma.
So comparison is difficult.

Cumulative accuracy Detlef44%
http://computer-go.org/pipermail/computer-go/2015-October/008031.html

Regards,
Hiroshi Yamashita


----- Original Message ----- 
From: "Detlef Schmicker" <ds2 at physik.de>
To: <computer-go at computer-go.org>
Sent: Wednesday, December 09, 2015 12:13 AM
Subject: [Computer-go] CNN with 54% prediction on KGS 6d+ data


> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi,
> 
> as somebody ask I will offer my actual CNN for testing.
> 
> It has 54% prediction on KGS 6d+ data (which I thought would be state
> of the art when I started training, but it is not anymore:).
> 
> it has:
> 1
> 2
> 3
>> 4 libs playing color
> 1
> 2
> 3
>> 4 libs opponent color
> Empty points
> last move
> second last move
> third last move
> forth last move
> 
> input layers, and it is fully convolutional, so with just editing the
> golast19.prototxt file you can use it for 13x13 as well, as I did on
> last sunday. It was used in November tournament as well.
> 
> You can find it
> http://physik.de/CNNlast.tar.gz
> 
> 
> 
> If you try here some points I like to get discussion:
> 
> - - it seems to me, that the playouts get much more important with such
> a strong move prediction. Often the move prediction seems better the
> playouts (I use 8000 at the moment against pachi 32000 with about 70%
> winrate on 19x19, but with an extremely focused progressive widening
> (a=400, a=20 was usual).
> 
> - - live and death becomes worse. My interpretation is, that the strong
> CNN does not play moves, which obviously do not help to get a group
> life, but would help the playouts to recognize the group is dead.
> (http://physik.de/example.sgf top black group was with weaker move
> prediction read very dead, with good CNN it was 30% alive or so :(
> 
> 
> OK, hope you try it, as you know our engine oakfoam is open source :)
> We just merged all the CNN stuff into the main branch!
> https://bitbucket.org/francoisvn/oakfoam/wiki/Home
> http://oakfoam.com
> 
> 
> Do the very best with the CNN
> 
> Detlef