Ballast is an important component of a railway track structure. It comes from the cruhing of rocks, and it ensures, from a mechanical point of view, the transmission of static and dynamic efforts induced by running of trains to the platform. The deterioration of the railway track under a large number of running of trains, in particular the settlement mechanism of the ballast layer or the lateral buckling of track, remains insufficienly known. In order to study the behavior of ballast layer a three-dimensional Discrete Element Method (DEM), based on the Non Smooth Contact Dynamic (NSCD) approach has been developed. Ballast grains are considered as rigid bodies with polyhedric shapes and the contact between grains is modeled by unilateral contact and dry Coulomb's friction laws. This model enable us to perform very specific investigations on railway track samples, with up to 30000 digitized grains, in order to understand local phenomenon and to propose rupture criterium.