There was some discussion about this a little while back:
Apache Answer does indeed look like a good option. But it’s more geared towards question/answer format over conversations. Also I don’t know about customizing authentication. But the big ones are mostly covered in that thread I linked to (I think!). 