Mode Ⅱ fracture initiation and propagation plays an important role under certain loading conditions in rock fracture mechanics. Under pure tensile, pure shear, tension-and compression-shear loading, the maximum Mode Ⅰ stress intensity factor, KI max; is always larger than the maximum Mode Ⅱ stress intensity factor, K Ⅱ max: For brittle materials, Mode Ⅰ fracture toughness, KIC; is usually smaller than Mode Ⅱ fracture toughness, KIIC: Therefore, KI max reaches KIC before K Ⅱ max reaches KIIC; which inevitably leads to Mode I fracture. Due to inexistence of Mode Ⅱ fracture under pure shear, tension-and compression-shear loading, classical mixed mode fracture criteria can only predict Mode Ⅰ fracture but not Mode II fracture. A new mixed mode fracture criterion has been established for predicting Mode Ⅰ or Mode Ⅱ fracture of brittle materials. It is based on the examination of Mode Ⅰ and Mode Ⅱ stress intensity factors on the arbitrary planey; KIðyÞ and KIIðyÞ; varying with yð 180 pypþ 180 Þ; no matter what kind of loading condition is applied. Mode Ⅰ fracture occurs when ðKII max=KⅠ maxÞo1 or 1oð K Ⅱ max=KI maxÞoðKⅡ C=KICÞ and KI max ¼ KIC at yIC: Mode Ⅱ fracture occurs when ðKⅡ max=KI maxÞ>ðKIIC=KICÞ and KⅡ max ¼ KIIC at yIIC: The validity of the new criterion is demonstrated by experimental results of shear-box testing. Shear-box test of cubic specimen is a potential method for determining Mode Ⅱ fracture toughness KIIC of rock since it can create a favorable condition for Mode Ⅱ fracture, i.e. KⅡ max is always 2-3 times larger than KⅠ max and reaches KⅡC before KⅠ max reaches KⅠC: The size effect on KⅡC for single-and double-notched specimens has been studied for different specimen thickness B; dimensionless notch length a=W (or 2a=W) and notch inclination angle a: The test results show that KⅡC decreases as B increases and becomes a constant when B is equal to or larger than W for both the single-and double-notched specimens. When a=W (or 2a=W) increases, KⅡC decreases and approaches a limit. The a has a minor effect on KIIC when a is within 65-75°: Specimen dimensions for obtaining a reliable and reproducible value of KIIC under shear-box testing are presented. Numerical results demonstrate that under the shear-box loading condition, tensile stress around the notch tip can be effectively restrained by the compressive loading. At peak load, the maximum normal stress is smaller than the tensile strength of rock, while the maximum shear stress is larger than the shear strength in the presence of compressive stress, which results in shear failure.