The Connection

The 100^th mitzvah in the count of the Chinuch is the negative commandment not to remove the choshen from the efod.  He explains that the commandment is not to remove the choshen from the efod when the kohan is serving in the mikdash.  It is clear from the Rambam (Kli Mikdash 9:10) that he disagrees and holds the lav applies always, even when not doing avodah and only if one destroys the connection between the choshen and efod, not merely separates them.  The Rambam holds that the lav is similar to the lav of destroying bigdah kehunah (see 28:32 and Rambam ibid law 3,) which only applies if its done in a destructive manner.  The chiddush of destroying the choshen from its connection to the efod is that even though they are two separate garments, it still is prohibited.  However, the Chinuch holds it is a law in avodah.  The lav is that the kohan must always wear the choshen together with the efod, he can’t wear them as two separate garments, hence he holds the prohibition only applies at the time of avodah and that any form of separation is prohibited.  

Why is there a prohibition of seperating the choshen from the efod? The Gemorah Arachin 16a says that the choshen atoned for wrong judgement and the efod atoned for avodah zarah.  It is this juxtaposition of בין אדם למקום ובין אדם לחבירו that the Torah wants to emphasize.  These aren't two independent branches of our religion, they are intertwined and go hand in hand.