Cayley hash functions are a family of cryptographic hash functions constructed from Cayley graphs, with appealing properties such as a natural parallelism and a security reduction to a clean, well-defined mathematical problem. As this problem involves non-Abelian groups, it is a priori resistant to quantum period finding algorithms and Cayley hash functions may therefore be a good foundation for post-quantum cryptography. Four particular parameter sets for Cayley hash functions have been proposed in the past, and so far dedicated preimage algorithms have been found for all of them. These algorithms do however not seem to extend to generic parameters, and as a result it is still an open problem to determine the security of Cayley hash functions in general. In this paper, we study the case of Chiu's Ramanujan graphs. We design a polynomial time preimage attack against the resulting Cayley hash function, showing that these particular parameters like the previous ones are not suitable for the construction. We extend our attacks on hash functions based on similar Cayley graphs as Chiu's Ramanujan graphs. On the positive side, we then suggest some possible ways to construct the Cayley hashes that may not be affected by this type of attacks. Our results contribute to a better understanding of the hard problems underlying the security of Cayley hash functions.